ARCH and REQUIRES_PYTHON plan

[henryiii]

1. We continue with CIBW_ARCHS as we are now; "auto" is the default, and auto (could) include universal2 for macOS, rather like it already includes 32bit for 64bit on the other OSs. I think most users will find this ideal for enabling or playing with the new architectures, since they are using things like cp36* now in build and that doesn't have to change unless they want to split based on it, see the final comments. macOS 11 considers "universal2" to be auto.
2. We add a CIBW_MIN_PYTHON flag, with the usual variants (and maybe command line flag? Don't know what the policy is for that). If it is not set, we read the value from PEP 621's location if set (eventually), and for now use 2.7 (27?) if not set. For CIBW 2.0, we can change the default to 3.6 (36?), but users could get at least 2.7 back by manually specifying this. This is mostly so we can keep the defaults "nice", while also supporting the full range that we can support. From 2.0 on, we raise the default minimum to always match the official Python EOL. feat: support brace expansion in BUILD/SKIP #527
3. (Optional) We upgrade to using the wcmatch library; this gives {} syntax, which I love using on the shell (partially because I'm a fish user and tab completion works correctly when using it, so it's perfect for moves and such). cp{27,35}* seems like it should work. Ve can activate the BRACE and NEGATE flags; this is simpler than my example above (basically just BUILD + SKIP), and is already well documented so we don't have to try to teach people how to use it, we can just point at this. Eventually, if this works well, we could push users away from the SKIP flag.
4. (Optional) We could add a little javascript selection in the docs as shown above. I just think that's very handy and will be "flashy" :)

ARCH/MIN_PYTHON are about telling cibuildwheel what you are interested in supporting, and is something we can adjust over time to "ideal" defaults, unless a user overrides them, in which case they specifically want that thing. With SKIP, which is in this category sort of, user overrides blend CPython/PyPy, Python version, and OS-ARCH, so we can't control each default separately without the new variables. BUILD is more about telling it what "this" job is trying to build out of the total allowed set.

Further; I propose we split the unreleased "special" value for "arch" into three special values:

"native": The native architecture (always one value).
"auto": Produces the native architecture + 32bit on 64bit non-macOS systems, or + universal2 on macOS. Basically "everything you can run tests on". This would be default, just like auto for CIBW_PLATFORM. I think this was first proposed by @mayeut ?
"all": Every known architecture, 3 archs on macOS, 2 on Windows, and 5 on linux, currently. Could grow in the future.

This makes universal2 fit in nicely, and provides a simple way to recover classic behavior. If you want to run on macOS 11 but want the old behavior, just set CIBW_ARCHS_MACOS: native. That would work nicely if you have separate runs on hypothetical Apple Silicon runners someday. You could also use native if you use the architecture setting on the setup-python action, which is currently non-trivial to implement (due to different spellings on Windows and Linux).

And "all" lets you put all the work on build selectors; users would have to be aware that the "all" set could grow in future versions if new manylinux archs were added.

Native example:

matrix:
  - os: [windows-latest, ubuntu-latest]
  - architecture: [x86, x64]
...
- uses: actions/setup-python@v2
  with:
    python-version: 3.8
    architecture: ${{ matrix.architecture }}

# What to put here for CIBW to avoid two copies of 32bit wheels?

This is probably best worked on after #484 goes in.

Originally posted by @henryiii in #516 (comment)

[henryiii]

@joerick and @YannickJadoul, feel free to edit the above as needed, tried to paste together something actionable. @mayeut feel free to commit. @Czaki too.

[YannickJadoul]

Just one minor question about 2: is that something we want to introduce now already, or only when 2.0 is in sight?

[henryiii]

I would introduce it now, so users who care can start explicitly pinning the min Python to 2.7. Maybe we could even print a warning if not pinned and an old wheel is built, saying it will change in 2.0 to defaulting to 3.6. I wouldn't change the default until 2.0, though, so bumping to 1.8 won't suddenly cause missing 2.7/3.5 wheels when manylinux still supports them.

Not that I really expect people to read warnings in CI logs. The folding at least helps.

I don't really know what @joerick plans for 2.0, basing this reasoning on SemVer.

[henryiii]

The days for PyPy 2 might be numbered after packaging drops support pypa/packaging#376 . Given they want to merge this before releasing a version that supports universal2 wheels, pypy2 might not be able to ever work properly on Apple Silicon.

[henryiii]

Minor change, now that I've played with packaging, let's make this CIBW_REQUIRES_PYTHON instead, and use packaging.selectors.SelectorSet. That will be identical to how project.requires-python will work, so it will be easier to set from that when we support it. So >=3.6 would be the default in 2.0, and >=2.7 will be the default now (we already don't support 3.0-3.4, so it's okay if they happen to match).

Sadly, setuptools spells this python_requires, but the Wheel metadata slot is Requires-Python, so that's the more "official" name and the one going forward. https://packaging.python.org/specifications/core-metadata/#requires-python

[joerick]

Just to chime in, this plan makes sense to me! The only thing really outstanding on this is (2), the CIBW_MIN_PYTHON option.

let's make this CIBW_REQUIRES_PYTHON instead, and use packaging.selectors.SelectorSet. That will be identical to how metadata.requires-python will work, so it will be easier to set from that when we support it. So >=3.6 would be the default in 2.0, and >=2.7 will be the default now (we already don't support 3.0-3.4, so it's okay if they happen to match).

Hmmmmmmmm I'm 50/50 on this idea, versus the CIBW_MIN_PYTHON approach. I worry it might make the option more expressive than we really need, potentially creating confusion between this option and BUILD/SKIP. Also, if we can pull compatibility from metadata.requires-python, I think I'd rather encourage users to put that info there instead of using our option at all. We could document our option as a way to 'change the default' Pythons built, but present metadata.requires-python as the 'right' way to do it.

[henryiii]

It will complicate the option to have both CIBW_MIN_PYTHON and project.requires-python both with different semantics (okay, now that I think about it, not by much at all, just adding >= should convert them). I could also see it drawing people into trying to use CIBW_MIN_PYTHON instead just because they like the syntax better or see it as simpler / more correct. I like having it literally just a stand-in for metadata.requires-python if someone doens't want to change pyproject.toml yet, or doesn't have a pyproject.toml, or needs to override an incorrect setting in pyproject.toml.

[henryiii]

Feel free to check out the docs example of #536; after writing the docs I'm moderately/strongly in favor of CIBW_REQUIRES_PYTHON. It is much more of a mental burden to have an override for project.requires-python with a totally different name and syntax (CIBW_MIN_PYTHON), that we don't really want to push people toward anyway; it's more of an escape hatch when you are building a project when don't have control over the pyproject.toml file (or refuse to add one). It's not because CIBW_REQUIRES_PYTHON is more expressive, it's the extra overload of having two different ways to do the same thing. Also, every project should already have at exactly this string in setup.py/setup.cfg under python_requires= already.

It's pretty obvious that CIBW_REQUIRES_PYTHON overrides project.requires-python, but I could see a user thinking that CIBW_MIN_PYTHON complements or combines with project.requires-python like BUILD/SKIP somehow.

Notice how we'd have to draw attention to a MIN_PYTHON setting, while this slips in as just another spelling for project.requires-python.

[joerick]

I see your points. The implementation/docs are neater if they share syntax. But I'm still concerned about confusion between this and BUILD/SKIP, and the name is confusing, because it reads 'cbuildwheel requires python', which is isn't right because it's the package that requires python, but even then it doesn't really require it in this context, etc. etc.

Perhaps there's a way to reframe this by changing the name? I was wondering about CIBW_DEFAULT_PYTHON_VERSIONS, but using the same syntax as project.requires-python. I kinda like the long option name, because it seems like more of a niche or unusual option compared to BUILD/SKIP. Also it implies that you're changing a default, but that the package contains the 'real' setting - project.requires-python should override this, if both are set.

But I must admit CIBW_MIN_PYTHON still seems like a simpler solution to me. The purpose is that is clear - we can change the default, but users can set it if they want to build something older. But, I concede, it's less clear how that would interact with project.requires-python when both are set.

[henryiii]

• How about CIBW_PROJECT_REQUIRES_PYTHON? That still makes it clear that it's an override for project.requires-python.
• A simpler solution isn't really simpler if there is already a more complex solution too, and people have to read both. Notices in the current docs, I don't have to discuss the variable at all, since it's simply the same as the item in the file. A new section would have to be added to explain MIN_PYTHON and how that relates. Also, simpler solutions tend to draw people in.
• If someone wanted to drop Python 3.5 support but not 2.7 support, CIBW_MIN_PYTHON isn't powerful enough to describe that. Again, I'm not all that worried, since we have a better solution (the metadata read), but something to keep in mind; and there are reasons someone might not want to add the file/setting yet, not the least of which is setuptools has not added support yet. The goal here is to use BUILD to pick what you want to build out of all valid build configurations, and to have invalid configurations (due to Python versions a package does not support) not even part of the equation.
  • Even the current solution isn't powerful enough to drop CPython without PyPy. I think that's a clear special case - if you support PyPy 2.7 but not CPython 2.7, you have to use SKIP to filter CPython 2.7 out, since you do support the Python 2.7 language, just not a specific instance (cp27-*), and that's a job for build selectors. @mattip could correct me, but I think PyPy follows the Requires-Python metadata setting.
• I think it is important to make it an override, as people sometimes have to build packages where they don't control the setup files. 99% of the time, the files should be right, but there are special cases. NumPy, IIRC, kept Python compatibility for one extra release after dropping it from Requires-Python. If you were building your own NumPy wheelhouse, you might want to override it. Especially for PyPy?...
• Just a wild thought; we could look in setup.cfg:python_requires if pyproject.toml does not contain it. It is clearly stated in the PEP that pyproject.toml:project settings is considered canonical and overrides all other files, so the ordering here is easy; it comes after. It would be technically easy to add with configparser, it would enable projects today to start using it without adding a pyproject.toml setting. In fact, I use that instead of setup.py for non-dynamic values already in most of my projects, so no change would be needed for those, cibuildwheel would simply stop listing python versions I don't support. I have no idea if this is a good idea, just thought of it.

[henryiii]

PS: The implementation is actually the easiest part, I'm worried about adding more complexity for users by having two ways to do this that are not the same, but not implementation.

Something like this:

requires_python = f'>={os.getenv("CIBW_MIN_PYTHON", "2.7")}'

If we were to drop 3.5 sooner than 2.7, this would be more complicated.