Put reasonable lower bounds on providers in Airflow's extras

[notatallshaw]

Description

Airflow should set lower bounds on the providers it specifies by extras.

Use case/motivation

Airflow provides a number of extras that point to providers, e.g. apache-beam points to the provider apache-airflow-providers-apache-beam , but in the metadata it does this completely unbounded:

Provides-Extra: apache-beam
Requires-Dist: apache-airflow-providers-apache-beam; (python_version != "3.12") and extra == 'apache-beam'

This leads to the problem that dependency resolution tools might backtrack to apache-airflow-providers-apache-beam==1.0.0 when trying to find solutions, which can cause resolution failures, e.g. astral-sh/uv#3078

I don't think the lower bounds need to even be particularly strong, just avoid letting resolution tools attempt extremely old versions of providers that are known to be problematic in some sense. In this example putting a lower bound of apache-airflow-providers-apache-beam>=2.0.0 would solve this issue.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[notatallshaw]

cc @potiuk digging deep into resolutions (astral-sh/uv#3078) I found that uv can end up aggressively checking older versions of (transitive) dependencies.

If airflow ever wanted to reccomend uv as a tool to user to install it, I would strongly reccomend adding some reasonable lower bounds on providers given by airflow's extras.

[potiuk]

Yes I think it might make sense to put some limits, but it's hard to say what those limits should be.

Generally speaking, the point of having old providers (even very old) and new Airflow is the whole point of separating providers and Airflow - simply, if provider has breaking changes, users can still upgrade Airflow but keep some providers not updated to latest versions.

The question is: "DEFINE REASONABLE". We already have some rules that we follow on providers supporting minimum Airflow versions - and those rules are well defined. https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow

If someone will come up with a proposal for what would be "REASONABLE", I think it makes sense - the extra limits are not hard - they are soft, so it does not even "block" the users if they want to use older versions.

If so - maybe a good idea will be to raise the proposal as discusison in our devlist - this is where we discuss such ideas - for example the 12 months policy for providers have been discussed in:

• https://lists.apache.org/thread/csczm7xmnntdz9wtjbod8pqgt13zoggo

and lazy consensus proposed and achieved here:

• https://lists.apache.org/thread/43v1dnww37t0o924mzo365ot2p58k864

Similarly - discussion should be started, with reasoning and proposal, and then after achieving a consensus, we should implement it. I am all for it and I am also super-happy to guide someone doing it. IT's been a long time I was doing all these things myself, but this time I started to think about others stepping in and driving this - I am happy to support and guide implementation of it so if you feel like it @notatallshaw -> I would be super-happy to take the back seat on that one.

[notatallshaw]

I am happy to put a well thought out proposal together on the mailing list, but beyond that I don't think I can either lead or implement it.

I'm pretty much at capacity with my OSS work, I already have a patch for Airflow I promised to turn into a PR last year and I've not managed to get round to doing (in part delayed because of company bureaucracy, in part I started spending a lot more focus on Python's packaging standards and it ate up all my OSS energy).

But if putting a proposal on the mailing list is a good start, without commiting to implementing it myself, I will do that soon.

[potiuk]

Sure. Let's start with that. Implementation might be simple and possibly someone else might volunteer - we already have a few people who know their ways around that code.