Can't install ansible-2.10.7 with ansible-operator:v1.26.1 & 1.27.0 on s390x platform while 1.26.0 works

[YanFenGuo]

Bug Report

What did you do?

I'm trying to install Ansible 2.10 into ansible-operator:v1.26.1.

What did you expect to see?

I want to install Ansible 2.10 into ansible-operator:v1.26.1, I can do this with ansible-operator:v1.26.0, but not 1.26.1.
I can also do this on x86 or ppc64le, only s390x platform doesn't work.

What did you see instead? Under which circumstances?

Launch Ansible Operator with below command:

docker run -u 0 -it --entrypoint="" --rm \
--name operator \
-e WATCH_NAMESPACE=ecm-fncm-s \
-e run="run --watches-file=./watches.yaml --ansible-verbosity 5" \
-e operator="/usr/local/bin/ansible-operator " \
quay.io/operator-framework/ansible-operator:v1.26.1 \
bash

Then execute below commands to update Ansible:

$pip3 uninstall -y ansible;
$python3.8 -m pip install --upgrade pip;
$yum install -y rust;
$pip3 install --no-cache-dir ansible==2.10.7

It failed with below error:

bash-4.4# pip3 install --no-cache-dir ansible==2.10.7
Collecting ansible==2.10.7
  Downloading ansible-2.10.7.tar.gz (29.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 29.9/29.9 MB 699.7 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting ansible-base<2.11,>=2.10.5
  Downloading ansible-base-2.10.17.tar.gz (6.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/6.1 MB 797.7 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/site-packages (from ansible-base<2.11,>=2.10.5->ansible==2.10.7) (3.1.2)
Requirement already satisfied: PyYAML in /usr/local/lib64/python3.8/site-packages (from ansible-base<2.11,>=2.10.5->ansible==2.10.7) (6.0)
Collecting cryptography
  Downloading cryptography-39.0.1.tar.gz (603 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 603.6/603.6 kB 814.3 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting packaging
  Downloading packaging-23.0-py3-none-any.whl (42 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 kB 1.0 MB/s eta 0:00:00
Requirement already satisfied: cffi>=1.12 in /usr/local/lib64/python3.8/site-packages (from cryptography->ansible-base<2.11,>=2.10.5->ansible==2.10.7) (1.15.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib64/python3.8/site-packages (from jinja2->ansible-base<2.11,>=2.10.5->ansible==2.10.7) (2.1.2)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/site-packages (from cffi>=1.12->cryptography->ansible-base<2.11,>=2.10.5->ansible==2.10.7) (2.21)
Building wheels for collected packages: cryptography
  Building wheel for cryptography (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for cryptography (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [181 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-s390x-cpython-38
      creating build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/__about__.py -> build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/exceptions.py -> build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/fernet.py -> build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/utils.py -> build/lib.linux-s390x-cpython-38/cryptography
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat
      copying src/cryptography/hazmat/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat
      copying src/cryptography/hazmat/_oid.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat
      creating build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/base.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/certificate_transparency.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/extensions.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/general_name.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/name.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/ocsp.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      copying src/cryptography/x509/oid.py -> build/lib.linux-s390x-cpython-38/cryptography/x509
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends
      copying src/cryptography/hazmat/backends/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings
      copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/_asymmetric.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/_cipheralgorithm.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/_serialization.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/padding.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/backends/openssl
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/openssl
      copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/openssl
      copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/openssl
      copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/openssl
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/types.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/asymmetric
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/ciphers
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/kdf
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      copying src/cryptography/hazmat/primitives/serialization/pkcs7.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/serialization
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/twofactor
      copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/twofactor
      copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/twofactor
      copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/primitives/twofactor
      running egg_info
      writing src/cryptography.egg-info/PKG-INFO
      writing dependency_links to src/cryptography.egg-info/dependency_links.txt
      writing requirements to src/cryptography.egg-info/requires.txt
      writing top-level names to src/cryptography.egg-info/top_level.txt
      reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      no previously-included directories found matching 'docs/_build'
      warning: no previously-included files found matching 'vectors'
      warning: no previously-included files matching '*' found under directory 'vectors'
      warning: no previously-included files matching '*' found under directory '.github'
      warning: no previously-included files found matching 'release.py'
      warning: no previously-included files found matching '.readthedocs.yml'
      warning: no previously-included files found matching 'ci-constraints-requirements.txt'
      warning: no previously-included files found matching 'tox.ini'
      warning: no previously-included files found matching 'mypy.ini'
      adding license file 'LICENSE'
      adding license file 'LICENSE.APACHE'
      adding license file 'LICENSE.BSD'
      adding license file 'LICENSE.PSF'
      writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
      copying src/cryptography/py.typed -> build/lib.linux-s390x-cpython-38/cryptography
      copying src/cryptography/hazmat/bindings/_openssl.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings
      creating build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      copying src/cryptography/hazmat/bindings/_rust/__init__.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      copying src/cryptography/hazmat/bindings/_rust/asn1.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      copying src/cryptography/hazmat/bindings/_rust/ocsp.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      copying src/cryptography/hazmat/bindings/_rust/pkcs7.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      copying src/cryptography/hazmat/bindings/_rust/x509.pyi -> build/lib.linux-s390x-cpython-38/cryptography/hazmat/bindings/_rust
      running build_ext
      generating cffi module 'build/temp.linux-s390x-cpython-38/cryptography.hazmat.bindings._openssl.c'
      creating build/temp.linux-s390x-cpython-38
      running build_rust
      /tmp/pip-build-env-4nb5oi_9/overlay/lib/python3.8/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'cryptography.hazmat.bindings._rust' as data is deprecated, please list it in `packages`.
          !!


          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'cryptography.hazmat.bindings._rust' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'cryptography.hazmat.bindings._rust' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'cryptography.hazmat.bindings._rust' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.


      !!

        check.warn(importable)
      cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features pyo3/abi3-py36 pyo3/extension-module -- --crate-type cdylib

          =============================DEBUG ASSISTANCE=============================
          If you are seeing a compilation error please try the following steps to
          successfully install cryptography:
          1) Upgrade to the latest pip and try again. This will fix errors for most
             users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
          2) Read https://cryptography.io/en/latest/installation/ for specific
             instructions for your platform.
          3) Check our frequently asked questions for more information:
             https://cryptography.io/en/latest/faq/
          4) Ensure you have a recent Rust toolchain installed:
             https://cryptography.io/en/latest/installation/#rust

          Python: 3.8.13
          platform: Linux-4.18.0-372.32.1.el8_6.s390x-s390x-with-glibc2.2
          pip: n/a
          setuptools: 67.4.0
          setuptools_rust: 1.5.2
          rustc: 1.62.1 (Red Hat 1.62.1-1.module+el8.7.0+16002+ac58477b)
          =============================DEBUG ASSISTANCE=============================

      error: Unable to execute 'cargo' - this package requires Rust to be installed and cargo to be on the PATH
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects
bash-4.4#

Pls let me know how I can make this work. Thank you.

[YanFenGuo]

ansible-operator:v1.27.0 has same issue.

[everettraven]

@YanFenGuo Thanks for raising this issue! We had a previous issue open that I believe is related to this (#6302) that we thought we had resolved but resulted in some build failures while we were trying to cut the v1.28.0 release.

I think this particular error you are receiving is the exact problem we were encountering in our build process and we had significant trouble resolving that error. I recently created a PR (#6348) to revert our changes that led to the creation of #6302. The plan as of now is to make sure this fix lands in the v1.28.0 release successfully and then create patch releases for v1.26 and v1.27 that fixes the problem in v1.26.1 and v1.27.0.

[caixiangibm]

@everettraven , the PR (#6348) reset cryptography = "==3.3.2" . While is it possible for ansible-operator to preset cryptography to 39.0.1 or higher to address vulnerability CVE-2023-23931 on all supported architectures (including amd64/s390x/ppc64le). I noticed there is active PR for that #6297 but no further updates.

[everettraven]

@caixiangibm #6348 was reverting a bump to cryptography 39.0.0 that was made in #6269 . This caused some issues in our CI resulting in our s390x and ppc64le builds of the ansible-operator no longer working properly because the cryptography package does not have a wheel built by default for these architectures. We attempted to try and update the build process to build the cryptography package within the image but that would consistently fail in our CI resulting in no images being built and we were unable to figure out why.

Due to this, our short-term plan is to stick with rolling back to cryptography 3.3.2 since that it what it was pinned to previously. We are still looking into and discussing longer term solutions.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[caixiangibm]

@everettraven , what about your long-term solution for cryptography version upgrade on s390x / ppc64le ?
It seems that we still have to cryptography 3.3.2 pinned currently?

[everettraven]

@caixiangibm Apologies for my delayed response! I just recently created a PR (#6483) that is a fairly large update to the ansible operator and dependencies. I included multi-stage builds in that PR in an attempt to make it so that these changes are also available on s390x and ppc64le architectures