Pip cannot download macosx_10_9_universal2 from Apple Silicon

[henryiii]

Pip on Apple Silicon does not consider a properly named Universal2 wheel macosx_10_9_universal2 as a valid candidate; the only universal wheel it will consider valid is macosx_11_0_universal2, which mostly defeats the purpose of being universal (especially for uses like bundling resources for executable packages). This is due to a bug in Packaging reported mid December pypa/packaging#379, that was quickly fixed pypa/packaging#380 and then merged at the beginning of the year.

I have opened an issue with packaging asking for a new release two weeks ago, pypa/packaging#385, but they wanted to wait till just before a Pip release so they wouldn't have to release twice (I think). Pip just released 21 and 20.3.4 without a new release of packaging, so this is still broken. I was asked to open an issue with Pip asking for the new release of Packaging (I think, I'm getting confused at this point...)

cibuildwheel is hoping to ship ARM/Universal2 support soon pypa/cibuildwheel#484, and the wheels produced will either have to be manually renamed with dual identifiers (macosx_10_9_universal2.macosx_11_0_universal2 ) as a hack, or they will not be downloadable on Apple Silicon with Pip.

Ideally, I think it would be nice to have a 20.3.5 and 21.0.1 with this fix, since cibuildwheel ships get-pip.py that then runs the pinned dependencies, and it supports the same set of Python that manylinux supports, so we would either have to ship two versions of get-pip (which is already large), or we'd have to do some sort of trickery on apple silicon to ensure user pins can access Universal wheels whenever ARM runners show up but still also have the option to pin Pip itself. (So no rush at all on the 20.3.5 ;) )

[pradyunsg]

puts on 20.3 release manager hat

I don't want to have overlapping release cycles, and the only reason I can see for having a 20.3.5 with this is for Python 2/3.5 -- which I really don't see much point in continuing support for at this point. It was already exceedingly annoying to cut 20.3.4, since the master branch has diverged significantly and adding vendoring changes on top is even worse.

I think it's better to consider 20.3 as a closed release cycle. Yes, this means that Python 2 and Python 3.5 have degraded experiences with the newest MacOS, but I'm fine with that -- they were EOL around/before the new MacOS was released.

Overall, I don't think this change justifies a 20.3.5.

takes off release manager hat

---

It's @pfmoore's call on 21.0.1. Based on my past experience from cutting bugfixes off master, I think it should be pretty straightforward and that we should do this -- I imagine that it'll depend on his availability though.

[henryiii]

was already exceedingly annoying to cut 20.3.4

We brought this up mid December, before the end of support and before either of these releases was cut. I explicitly opened an issue in Packaging asking for a release to try to make sure this didn't happen. I don't know what else I could have done.

I understand dropping Python support is fun, but it seems like it's being prioritized over supporting Apple Silicon?

Yes, this means that Python 2 and Python 3.5 have degraded experiences with the newest MacOS, but I'm fine with that

I'm not worried about this; macOS 11 Intel already does not support 3.5 and ARM does not support anything before 3.9 (maybe soon 3.8); I'm worried about shipping multiple bootstrap scripts. There's no single "initial" version of get-pip that can run on all versions of Python cibuildwheel (versions supported based on manylinux) supports, and also can get Universal2 wheels on ARM. It would be nice to be able to ship one script until 2.7/3.5 are dropped from manylinux (which I assume is when we drop support too). We currently use 19.something but will need to upgrade to 20.3.x when macOS 11 runners start becoming common, I would ideally like to upgrade once to 20.3.5. This likely will also be needed for PyPy2, as well.

This part is not required, but it would be very nice. We could add a hack that does a pip upgrade to one of two versions (macOS 11 vs. other) after bootstrapping but before running pip again, etc. Also, this is almost entirely used to get standard things (pip, wheel, and so forth) that are pure wheels, not Universal2, so likely is not critical. And by the time Apple Silicon runners become common, maybe manylinux will have dropped support, and that likely means cibuildwheel will have to, too.

---

21.0.1 is a high priority for me, though, I'd love to see anything that can download properly named Universal wheels on Apple Silicon!

PS: Apologies if I sound a bit grumpy, I'm just disappointed because I tried to help this get resolved, and I really don't want to ship weird rename hacks for universal2 wheels...

[pfmoore]

Thanks @pradyunsg.

Based on my past experience from cutting bugfixes off master, I think it should be pretty straightforward and that we should do this

The documented release process only covers XX.Y from master and XX.Y.Z+1 from a branch with commits being cherry-picked. That concerns me a lot, because if I'm not working from a documented process, we have a certain level of risk. And my time is limited - if I screw up the release by trying to modify the process on the fly, I won't have time to fix it. So if you have time to write up how to do a bugfix release direct from master, that would be useful.

However, at the moment the ball is in packaging's court. Until they release, this discussion is irrelevant. I can make some time next weekend to do a 21.0.1 (given the above comments), but I have no idea if there will be a packaging release this week, to allow for that.

Personally, I don't have any experience with the Mac platform, so I can't judge how significant this limitation is. My understanding is that older versions of pip didn't have this support, though, so it's a "delay in a new feature" rather than a regression, which makes me feel that we should be cautious in rushing a release just to include it. But I'll take advice on this from other pip maintainers who know the Mac platform issues better than I do.

I'm getting confused at this point...

So am I, to be honest. You seem to be saying that packaging deliberately delayed a release that is ready to go, which seems very odd. Conversely, pip's release schedule is documented, and 21.0 was on schedule, so it doesn't seem like the timing should have been a surprise to anyone. But ultimately, it's not really that important now - whatever communication breakdown happened, we understand the situation now.

I suggest we wait for packaging to release a new version. At that stage, we can update pip's vendored version, and look at whether I can make time to do a follow-up release (or whether another pip maintainer is available to do so).

[pfmoore]

Apologies if I sound a bit grumpy, I'm just disappointed because I tried to help this get resolved, and I really don't want to ship weird rename hacks for universal2 wheels...

Understood. However, we're all volunteers on pip and packaging as well, and we try very hard to not need tight co-ordination, precisely because it messes around with people's free time. So it's also frustrating for me to find out that there was some sort of discussion going on in a channel that I wasn't aware of, that impacted a release I was doing with very little available time, and I now feel somehow responsible for not having found out about it in advance 🙁

[henryiii]

My understanding is that older versions of pip didn't have this support, though, so it's a "delay in a new feature" rather than a regression

It's somewhere in-between. The plan was to have Universal2 wheels the recommendation for ARM support for now (CC @ronaldoussoren), up to the point there was even a suggestion of prioritizing them over native wheels (pypa/packaging#381, which I was not in favor of). So pypa/packaging#319 was added around 20.5-20.8. But there's a bug, it doesn't consider anything before 11.0 a "matching" universal wheel. So if you make a wheel with a macosx_10_9_universal2 tag (or any other number that's not 11_0), it will not load. Normally, it is supposed to match any earlier number, and it does on Intel. That's the bug that was fixed. So it's a bug fix in a feature that was included already and shipped that kept anything from matching except 11_0. (in Pip 20.3)

I now feel somehow responsible for not having found out about it in advance

I don't think it was anyone's "fault", just communication issues, and a misunderstanding of the connection between packaging and pip on my part.

[henryiii]

@brettcannon released Packaging 20.9 with the fix for Univeral2 tag names! 👏

[pfmoore]

Because of #9462, I can't do the re-vendoring without handling it manually, and I have limited time this weekend. If someone else can prepare the vendoring PR, it will significantly increase the likelihood of getting a 21.0.1 release this weekend.

[henryiii]

Targeting master, right? I guess it can be redirected if not.

[pradyunsg]

Closing since this is fixed now.