Pip try to install twice the same package at different versions, leading to conflicts

[Conchylicultor]

Description

It seems in some cases, Pip try to install multiple versions of the same package, leading to conflict resolution:

ERROR: Cannot install sphinx-apitree because these package versions have conflicting dependencies.

The conflict is caused by:
    myst-nb 0.4.0 depends on myst-parser~=0.7.1
    myst-nb 0.3.0 depends on myst-parser~=0.6.0
    myst-nb 0.2.2 depends on myst-parser~=0.6.0
    myst-nb 0.2.1 depends on myst-parser~=0.6.0
    myst-nb 0.2.0 depends on myst-parser~=0.6.0

It's not clear why pip install all those versions. Installing pip install myst-nb or pip install sphinx-apitree[ext] works. It seems the issue is triggered only when installing locally pip install .[docs].

Expected behavior

Packages are installed correctly.

pip version

24.3.1

Python version

3.11

OS

Ubuntu

How to Reproduce

# Optional, create virtual environment to isolate deps
pyenv virtualenv 3.11 pip_issue_repro
pyenv activate pip_issue_repro


# Reproduction
pip install kauldron[docs]

Or:

git clone https://github.com/google-research/kauldron
cd kauldron/
pip install .[docs]

Output

Output: https://gist.github.com/Conchylicultor/741379767180cb09a85b4433958719f6

The .[docs] only install sphinx-apitree[ext]: https://github.com/google-research/kauldron/blob/d409d54e9bfb9e5e5f2f2c70946ff7783dd098ca/pyproject.toml#L78

[project.optional-dependencies]
docs = [
    "sphinx-apitree[ext]",
]

If I install pip install sphinx-apitree[ext] directly, it's working. So not sure why pip fail to install when the same package is installed inside project.optional-dependencies

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for reporting. This issue is a duplicate of #12768 and #12317, pip isn't actually trying to install different versions of the same package, it's just a bug in the resolution and the final resolution impossible message. It will be fixed by #13001.

Looking at the resolution details the problem is largely about finding non-conflicting sphinx requirements, as myst_parser and MyST_NB often have contradictory requirements. Right now you could add pin your Sphinx version, e.g. Sphinx==8.1.3 and it should resolve.

However, this is still quite a problematic resolution and once that bug is fixed pip will struggle to resolve it. There are no reasonable lower bounds on the dependencies of this project: https://github.com/google-research/kauldron/blob/main/pyproject.toml#L17 and therefore the search space is massive, so I would strongly advise you add reasonable lower bounds to the requirements where you can.

I also note that even uv can not solve this resolution (uv pip install --dry-run kauldron[docs]==1) without allowing pre-releases (uv is more strict about pre-releases than the spec).

If I install pip install sphinx-apitree[ext] directly, it's working. So not sure why pip fail to install when the same package is installed inside project.optional-dependencies

Pip only installs what you ask it to install, it does not not guarantee it will keep consistent with the current environment, so when you give it a small set of requirements to install it is much faster.

[Conchylicultor]

Thanks for the answer. This makes sense

Right now you could add pin your Sphinx version, e.g. Sphinx==8.1.3 and it should resolve.

I think pinning dependencies would just add to the problem as if other packages are doing this, it makes finding compatible version even harder. In general I think it's best for libraries to have requirements as loose as possible. I assume if MyST_NB and myst_parser didn't pinned the deps, I wouldn't have encountered this to begin with.

But indeed just specifying a lower bound is good.

[notatallshaw]

I think pinning dependencies would just add to the problem as if other packages are doing this, it makes finding compatible version even harder. In general I think it's best for libraries to have requirements as loose as possible. I assume if MyST_NB and myst_parser didn't pinned the deps, I wouldn't have encountered this to begin with.

Yes, this is true in general, but given this is for a "docs" extra, I assumed that the only time someone would want this extra is when generating docs for the project. For such a case I think it's okay to pin.

FYI, I've added this requirement to a list of known problematic scenarios: https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks/blob/main/scenarios/problematic.toml#L240.

And you can see a representation of the resolution here: https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks/blob/main/output/problematic/kauldron-docs/24.3.json

The format of the resolution representation isn't final yet, so it may change in the future. But currently in the "added" section of each round you can see what packages it tried to add and the requirements those packages had, "pinned" is if something was successfully included in the resolution, and in the "rejected" section you can see the requirements that were involved when something couldn't be "pinned".

I'm working on seeing if it's possible to make it easy to identify where in a resolution things "went wrong" and if there are better heuristics pip could have to make a better resolution. Pip employs a fairly simple resolution strategy right now with a few optimizations, but it could definitely be better.