Poetry not always find latest version from PyPi

[valbendan]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.1
• Poetry version: 1.1.4

I have no idae with how poetry cache or pypi index works, hence this may be a not bug :)

Issue

Let's say I have two project named p1, p2.

p1 is published on pypi and p2 dependent on p1.

When a new version released for p1(let's say v2.0), I update dependent p1 = ">=2.0,<3.0" to my p2 project.

Then poetry update will complain:

SolverProblemError

    Because p2 depends on p1 (>=2.0,<3.0) which doesn't match any versions, version solving failed.

but poetry run pip install p1==v2.0 works.

If wait some time (about 5 ~30 minutes) after I release p1, poetry update finally will works.

PS: I DO NOT use pypi mirror.

[pcarbonn]

Running poetry update solved my problem.

[jeanphilippemercier]

I faced the same issue. The only way I found to get it to work is to clear the PyPI cache:

poetry cache clear pypi --all

then run poetry update.

Resolving dependency can take several 100 seconds.

[BartlomiejSkwira]

I also had to update pip to get the latest Black 21.5b1

python -m pip install --upgrade pip

[mlissner]

I call this art piece, "A portrait of a sad programmer". Note that it includes time stamps of me trying to install version 0.9.13 of a dep I just released. TL;DR is that the answer above works, but arriving there is painful:

eyecite-o10j3129-py3.8) 08:55:44::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update
(eyecite-o10j3129-py3.8) 08:55:54::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update
(eyecite-o10j3129-py3.8) 08:56:03::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update
(eyecite-o10j3129-py3.8) 08:56:06::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update

Perhaps exiting the poetry shell will clear some cache?

(eyecite-o10j3129-py3.8) 08:56:19::mlissner@gabbro::~/Programming/eyecite 
↪ exit
exit
08:56:24::mlissner@gabbro::~/Programming/eyecite 
↪ poetry shell
Spawning shell within /home/mlissner/.cache/pypoetry/virtualenvs/eyecite-o10j3129-py3.8
. /home/mlissner/.cache/pypoetry/virtualenvs/eyecite-o10j3129-py3.8/bin/activate
08:56:26::mlissner@gabbro::~/Programming/eyecite 
↪ . /home/mlissner/.cache/pypoetry/virtualenvs/eyecite-o10j3129-py3.8/bin/activate
(eyecite-o10j3129-py3.8) 08:56:26::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update
(eyecite-o10j3129-py3.8) 08:56:37::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

No dependencies to install or update

OK, I'll give it a few minutes:

(eyecite-o10j3129-py3.8) 09:03:04::mlissner@gabbro::~/Programming/eyecite 
↪ poetry remove courts-db
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing courts-db (0.9.11)
(eyecite-o10j3129-py3.8) 09:03:18::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.11 for courts-db

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing courts-db (0.9.11)

Arg. I guess I better google it. After finding this ticket:

(eyecite-o10j3129-py3.8) 09:03:23::mlissner@gabbro::~/Programming/eyecite 
↪ poetry cache clear pypi --all
Delete 886 entries? (yes/no) [no] yes
(eyecite-o10j3129-py3.8) 09:05:14::mlissner@gabbro::~/Programming/eyecite 
↪ poetry add courts-db@latest
Using version ^0.9.13 for courts-db

Updating dependencies
Resolving dependencies... (40.2s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating courts-db (0.9.11 -> 0.9.13)

Ahhhh.

[valbendan]

poetry cache clear pypi --all do the trick.

[radusuciu]

Also hit this today and the incantation posted above worked, thank you!

This happened to a package of mine that I had just updated - and published to pypi using poetry.

My poetry version is 1.1.6. I'll try and update and see if this still occurs when I release another update.

Edit: today I encountered the same issue with 1.1.6. Shortly thereafter I upgraded to 1.1.8, and the updated package was picked up.
Edit 2: I just pushed another version and poetry update again failed to find it, this time using 1.1.8. I suspect that in my previous edit the time delay between my attempt with 1.1.6 and then 1.1.8 was involved.

[kornpow]

Also had this problem today... I wish only deleting the cache would solve the problem, that would be understandable.

poetry remove {my-recently-updated-package}
poetry cache clear pypi --all
poetry add {my-recently-updated-package}@latest

[kakyoism]

poetry cache clear pypi --all do the trick.

This does not always work, either.

[kakyoism]

Also had this problem today... I wish only deleting the cache would solve the problem, that would be understandable.

poetry remove {my-recently-updated-package}
poetry cache clear pypi --all
poetry add {my-recently-updated-package}@latest

This seems to be the only way out.

[adriangb]

I've run into this as well. Cache seems to be too aggressive. Maybe failure to find a solution can trigger a cache refresh?

[rr-]

Just switched to poetry and I ran into this on my first day of using it. I agree that the cache should be wiped in case the package is not found.

[thclark]

My story

Yeah, I'm in total agreement that in the event of a solver error, poetry should rub cache and try again... however, I'm wondering if the actual issue is a bit more subtle - perhaps this points to a bug in the caching logic.

At the time of writing (6 May 2022), I'm experiencing this solver error (click to expand) when `sphinx-charts==0.1.2` package has been published since 12 Feb 2022.

 > [6/6] RUN poetry install --extras docs --no-ansi --no-interaction --no-root: 
#10 0.322 Skipping virtualenv creation, as specified in config file.            
#10 0.559 Installing dependencies from lock file                                
#10 0.768 Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.
#10 0.833 
#10 0.833   SolverProblemError
#10 0.833 
#10 0.833   Because power-curve-utilities depends on sphinx-charts (0.1.2) which doesn't match any versions, version solving failed.

More to the point, I'm experiencing it while building a docker container. Why is that important? Because the starting image layer has no cache in the first place.

Here's the dockerfile (click to expand).

# This is basically the Microsoft vscode devcontainer for python, no poetry yet
FROM windpioneers/gdal-python:familiar-catshark-gdal-2.4.1-python-3.9-dev

# Switch to vscode user
USER vscode
WORKDIR /workspaces/power-curve-utilities

# Install poetry
RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
ENV PATH "/home/vscode/.poetry/bin:$PATH"
RUN poetry config virtualenvs.create false

# Install python dependencies. Note that poetry installs any root packages by default,
# But this is not available at this stage of caching dependencies. So we do a dependency-only install here
# to cache the dependencies, then a full poetry install post-create to install any root packages.
COPY pyproject.toml poetry.lock ./
RUN poetry install --extras docs --no-ansi --no-interaction --no-root

It's the last RUN poetry install line that produces the solver error, But its a fresh install of poetry... it shouldn't have a cache at all.

I solved the issue by commenting out that last line, shelling into the container and doing as advised above:

poetry remove sphinx-charts                           
poetry cache clear pypi --all
poetry add sphinx-charts@latest

If I then uncomment the RUN poetry install line and rebuild the container again, it works fine. The significance of this is that the only things that could have changed in between failure and success are:

• any online cache or resources that got refreshed or updated (i.e. not my local cache, but something on a server)
• my pyproject.toml and poetry.lock files (copied in the prior image layer)

Hypothesis

I might be wrong, but I don't think anything online would have changed (e.g. the response from any query to pypi), because that last package was published months ago...

So I'm wondering if to set off this sequence of events, I had made some mistake (or there was some other problem) that put my lockfile out of sync with pyproject.toml... and whether that then gives a SolverError instead of barfing at me and saying Your poetry.lock file might be out of date, run 'poetry update sphinx-charts'.

I note that all of the solutions (eg from @kornpow and @kakyoism ) in this thread - with the exception of the trace provided by @mlissner - would also have refreshed any change in pyproject.lock, as well as refreshing the cache. @mlissner waited a few minutes, but his entire trace spanned <9 minutes and I don't know how long the cache expiry is in poetry but I bet it's at least 10 minutes, which would explain that experience.

(possible) Conclusion

My suspicion is that to solve this:

• if it's been only a few minutes since publishing the dependency, do a cache bust
• if it's been a long time and you get this error, do poetry update <problematic-library>
• perhaps @maintainers could provide a perspective, because maybe that SolverError could include a check that the dependency doesn't need updating? (maybe it does already and I missed it amid the volume of the logstream?)