ppc64le wheel

[ax3l]

Hi,

I am trying to upload my new ppc64le wheels with twine and it looks like installing twine pulls cryptography as a dependency.

That pull failed because it could not find a ppc64le wheel for cryptography and then fell back to a from-source build (This package requires Rust >=1.41.0.).

Do you think it's possible to provide alongside the aarch64 linux wheels also ppc64le wheels for cryptography?

[alex]

We only provide wheels for platforms we can test, and there's no acceptable CI providers for ppc64 (where acceptable means performance, capacity, UX, etc. -- I know travis has added ppc support, but they're not usable for OSS projects at this point IMO). Until there's ppc CI, there won't be wheels for it.

[reaperhulk]

twine generally does not depend on cryptography either. It likely only ever depended on it transitively through pyOpenSSL + requests, which is only relevant if you're on a very old Python 2.7.8 or below installation where the stdlib ssl module was not sufficient. However, even if an older twine did explicitly pull the dep the latest does not (as I just verified).

[ax3l]

Hi @alex,

there's no acceptable CI providers for ppc64 (where acceptable means performance, capacity, UX, etc. -- I know travis has added ppc support, but they're not usable for OSS projects at this point IMO).

Luckily that changed recently. arm64 and ppc64le are partner platforms on Travis and thus again fully free for open source projects (no credits needed): https://docs.travis-ci.com/user/multi-cpu-architectures/#partner-queue-solution

So you could use those for CI and CD :)

Alternatively, cibuildwheel supports cross-compilation on GitHub Actions and other free CI services via qemu. Admittedly, this can be a bit slow, depending on how much you need to build.

twine generally does not depend on cryptography either.

Hi @reaperhulk, I am not sure about this, it definitely gets pulled in the manylinux images. See this (demonstrated with x86_64, but same with ppc64le):

docker run -it quay.io/pypa/manylinux_2_24_x86_64

root@c44f38c1d034:/# python3.6 --version
Python 3.6.13
root@c44f38c1d034:/# python3.6 -m pip install -U pip
Requirement already satisfied: pip in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (21.1.2)
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
root@c44f38c1d034:/# python3.6 -m pip install -U twine
Collecting twine
  Downloading twine-3.4.1-py3-none-any.whl (34 kB)
Collecting keyring>=15.1
  Downloading keyring-23.0.1-py3-none-any.whl (33 kB)
Collecting colorama>=0.4.3
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting readme-renderer>=21.0
  Downloading readme_renderer-29.0-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: importlib-metadata>=3.6 in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (from twine) (4.3.1)
Collecting rfc3986>=1.4.0
  Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting tqdm>=4.14
  Downloading tqdm-4.61.0-py2.py3-none-any.whl (75 kB)
     |████████████████████████████████| 75 kB 2.8 MB/s 
Collecting requests>=2.20
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 5.6 MB/s 
Collecting pkginfo>=1.4.2
  Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 2.3 MB/s 
Requirement already satisfied: zipp>=0.5 in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (from importlib-metadata>=3.6->twine) (3.4.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (from importlib-metadata>=3.6->twine) (3.10.0.0)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 3.3 MB/s 
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting docutils>=0.13.1
  Downloading docutils-0.17.1-py2.py3-none-any.whl (575 kB)
     |████████████████████████████████| 575 kB 10.4 MB/s 
Collecting Pygments>=2.5.1
  Downloading Pygments-2.9.0-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 19.2 MB/s 
Collecting bleach>=2.1.0
  Downloading bleach-3.3.0-py2.py3-none-any.whl (283 kB)
     |████████████████████████████████| 283 kB 21.1 MB/s 
Collecting six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: packaging in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine) (20.9)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 13.4 MB/s 
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 6.6 MB/s 
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 25.5 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
     |████████████████████████████████| 145 kB 18.9 MB/s 
Collecting cryptography>=2.0
  Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 18.4 MB/s 
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
     |████████████████████████████████| 401 kB 38.1 MB/s 
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 18.2 MB/s 
Requirement already satisfied: pyparsing>=2.0.2 in /opt/_internal/cpython-3.6.13/lib/python3.6/site-packages (from packaging->bleach>=2.1.0->readme-renderer>=21.0->twine) (2.4.7)
Installing collected packages: pycparser, cffi, webencodings, urllib3, six, jeepney, idna, cryptography, chardet, certifi, SecretStorage, requests, Pygments, docutils, bleach, tqdm, rfc3986, requests-toolbelt, readme-renderer, pkginfo, keyring, colorama, twine
  WARNING: The script chardetect is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pygmentize is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script tqdm is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pkginfo is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script keyring is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script twine is installed in '/opt/_internal/cpython-3.6.13/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Pygments-2.9.0 SecretStorage-3.3.1 bleach-3.3.0 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 colorama-0.4.4 cryptography-3.4.7 docutils-0.17.1 idna-2.10 jeepney-0.6.0 keyring-23.0.1 pkginfo-1.7.0 pycparser-2.20 readme-renderer-29.0 requests-2.25.1 requests-toolbelt-0.9.1 rfc3986-1.5.0 six-1.16.0 tqdm-4.61.0 twine-3.4.1 urllib3-1.26.5 webencodings-0.5.1
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

(I see the same dependency pull for CPython 3.7-3.9 in that image.)

It looks like I am not the only one experiencing that issue, e.g. the cmake package distribution for PyPI also works around this by using an older release: https://github.com/scikit-build/cmake-python-distributions/blob/4730aeee240917303f293dffc89a8d8d5a4787c4/requirements-deploy.txt (I followed that work-around now to get my packages published from CI.)

[reaperhulk]

Looks like the dependency comes from secretstorage, which comes from keyring. Making keyring an optional dep for twine might be appropriate here, although that's quite possibly challenging for other reasons.

However, even when we had TravisCI we disabled ppc64le due to poor performance (added in #5192 and then removed in #5230). The unfortunate reality here is that this is a project with two maintainers and for our own sanity we've chosen to constrain the set of platforms we directly test against to keep our CI reliability high and our cycle time low. Given the need to prioritize we won't be re-adding it at this time, sorry.

You, of course, can choose to build this yourself, although it is admittedly somewhat involved on older platforms that don't have a supported OpenSSL.

[ax3l]

I have to admit, I even had trouble with Ubuntu 20.04 to get the crypography toolchain up on Travis for PPC64le... 😅 (I just need to support relatively recent platforms, i.e. RHEL8+ & recent Ubuntu with the recent manylinux images.)

ppc64le (contrary to S390X) is relatively popular in scientific computing and HPC at the moment, wheels for that would be quite useful. Should we talk to the twine people to get crypography out of the dependency chain somehow? Otherwise we might be blocking a relatively large downstream stack to upload ppc64le wheels, I fear.

Either way, thanks a lot for your work on cryptography. I understand that such decisions on what platform to support and test on are not easy from a maintenance viewpoint.

[reaperhulk]

I think filing an issue (or contributing a PR) for making keyring an optional dep of twine might be a worthwhile effort. That would remove the cryptography dependency, which isn't necessary for your use case. However, that's a behavioral change since twine installs that by default now and it'd need to become an extra like twine[keyring] or similar.

Ultimately we'll continue to strive to make our builds as simple as is feasible given the need for OpenSSL, a C compiler, and a rust compiler. And if ppc64le CI can get to parity with the performance of our aarch64 builders we'll consider re-adding it.