Unable to `poetry install` twisted trunk and treq

[DMRobertson]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 35
• Poetry version: 1.1.12
• Pyproject.toml:

  [tool.poetry]
  name = "test"
  version = "0.1.0"
  description = ""
  authors = ["David Robertson <david.m.robertson1@gmail.com>"]
  
  [tool.poetry.dependencies]
  python = "^3.10"
  Twisted = {git = "https://github.com/twisted/twisted.git", branch = "trunk"}
  treq = "^22.2.0"
  
  [tool.poetry.dev-dependencies]
  
  [build-system]
  requires = ["poetry-core>=1.0.0"]
  build-backend = "poetry.core.masonry.api"

Issue

I tried to install twisted trunk and treq as follows:

• poetry init
• poetry add git+https://github.com/twisted/twisted.git#trunk
• poetry add treq

Step three tried to install treq ^22.2.0 which is the newest version. This failed: see add treq.txt.

I find the output hard to read (not helped by Twisted's date-based versioning scheme), but I think what's happening is:

• treq 22.2.0 requires "Twisted[tls] >= 18.7.0"
• poetry deduces the set of possible twisted versions: Thus, treq (>=22.2.0,<23.0.0) requires twisted (18.7.0 || 18.9.0 || 19.2.0 || 19.2.1 || 19.7.0 || 19.10.0 || 20.3.0 || 21.2.0 || 21.7.0 || 22.1.0 || 22.2.0).
• This doesn't include include twisted 22.2.0.post0, the trunk version; I'm guessing this isn't being considered and that's why resolution fails?

I've also tried marking both with allow_prereleases=true but that didn't seem to change any behaviour---I'm guessing this refers to prereleases on PyPI rather than from-git prereleases.

Am I doing something wrong here?

[Secrus]

Did you try installing Twisted from PyPI not from git? Looking at file you provided, My guess would be that post0 confuses poetry solver to treat 22.2.0.post0 as lower version than 22.2.0 that would match requirements of treq.

[abn]

TL;DR you can workaround this by using the following add command for twisted instead.

poetry add --extras tls git+https://github.com/twisted/twisted.git#trunk

The root cause of this behaviour is the retrieval of twisted[tls] from the locked repository.

poetry/src/poetry/mixology/version_solver.py

Lines 359 to 367 in cf8999f

	locked = self._get_locked(dependency)
	if locked is None or not dependency.constraint.allows(locked.version):
	try:
	packages = self._provider.search_for(dependency)
	except ValueError as e:
	self._add_incompatibility(
	Incompatibility([Term(dependency, True)], PackageNotFoundCause(e))
	)
	return dependency.complete_name

As the tls extra was not activated the solver determines that the package twisted (locked) is not he same as twisted[tls] required.

poetry/src/poetry/mixology/version_solver.py

Lines 456 to 457 in cf8999f

	if not dependency.is_same_package_as(locked):
	return None

The complete name check fails.

https://github.com/python-poetry/poetry-core/blob/06f1f326eb512ec836b7f6379851d5fe2ea6eb30/src/poetry/core/packages/specification.py#L75-L76

This leads to the providers (ie. indexes) searched again for the right package with the extra tls. And as the project requires twisted==22.2.0.post0 (a version not available in PyPI, we fail to solve for treq's dependency on twisted[tls].

Think there are a few improvements to be made here. The same package check from core needs to be improved to handle extras.

[callahad]

Workaround works. Suggest we can close this?

[abn]

Think there is a root cause here that we should address, might leave this open so we can get that fixed.

[radoering]

As the tls extra was not activated the solver determines that the package twisted (locked) is not he same as twisted[tls] required.

@abn This sounds a bit like #4618

@DMRobertson @callahad Can you try if the issue is still present in poetry 1.2.0b1?

[abn]

Looks like adding treq still fails. But atleast now it states clearly it is a solver bug.

$ poetry add treq
Using version ^22.2.0 for treq

Updating dependencies
Resolving dependencies... (33.2s)

  RuntimeError

  [BUG] not twisted (22.2.0.post0 git rev trunk) is not satisfied.

  at ~/workspace/python-poetry/poetry/src/poetry/mixology/partial_solution.py:206 in satisfier
      202│             # As soon as we have enough assignments to satisfy term, return them.
      203│             if assigned_term.satisfies(term):
      204│                 return assignment
      205│ 
    → 206│         raise RuntimeError(f"[BUG] {term} is not satisfied.")
      207│ 
      208│     def satisfies(self, term: Term) -> bool:
      209│         return self.relation(term) == SetRelation.SUBSET
      210│ 

[radoering]

The issue is not limited to locked repository (poetry lock --no-update or poetry add with existing lock file). It can be triggered without an existing poetry.lock just by running poetry lock with the given pyproject.toml. #5770 seems to solve all of these use cases.