VCS dependency with extras requires explicit branch (due to double clone)

[neersighted]

• Poetry version: 1.2.2
• Python version: 3.10.8
• OS version and name: Arch Linux, Windows 10/11, macOS 13

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

When using a VCS dependency with extras and the Dulwich Git client, an explicit branch must be provided. The following pyproject.toml will fail to clone:

[tool.poetry]
name = "extra-second-clone"
version = "0.1.0"
description = ""
authors = ["Bjorn Neergaard <bjorn@neersighted.com>"]
readme = "README.md"
packages = [{include = "extra_second_clone"}]

[tool.poetry.dependencies]
python = "^3.10"
poetry-core = {git = "https://github.com/python-poetry/poetry-core", extras = ["demo"]}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

$ poetry lock -vvv
Loading configuration file /home/neersighted/.config/pypoetry/config.toml
Loading configuration file /home/neersighted/.config/pypoetry/auth.toml
Trying to detect current active python executable as specified in the config.
Found: /home/neersighted/.local/share/pyenv/versions/3.11.0/bin/python
Virtualenv extra-second-clone-IgA_y9qo-py3.11 already exists.
Using virtualenv: /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: extra-second-clone is 0.1.0
   1: derived: extra-second-clone
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /python-poetry/poetry-core/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/python-poetry/poetry-core at 'HEAD' to /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11/src/poetry-core
   1: fact: extra-second-clone depends on poetry-core[demo] (1.3.2)
   1: selecting extra-second-clone (0.1.0)
   1: derived: poetry-core[demo] (1.3.2) @ git+https://github.com/python-poetry/poetry-core
   1: fact: poetry-core[demo] (1.3.2) depends on poetry-core (1.3.2)
   1: selecting poetry-core[demo] (1.3.2 b0b1823)
   1: derived: poetry-core (1.3.2) @ git+https://github.com/python-poetry/poetry-core@HEAD
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /python-poetry/poetry-core/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/python-poetry/poetry-core at 'HEAD' to /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11/src/poetry-core
   1: Version solving took 1.310 seconds.
   1: Tried 1 solutions.

Failed to clone https://github.com/python-poetry/poetry-core at 'HEAD', verify ref exists on remote.

The following pyproject.toml will succeed:

[tool.poetry]
name = "extra-second-clone"
version = "0.1.0"
description = ""
authors = ["Bjorn Neergaard <bjorn@neersighted.com>"]
readme = "README.md"
packages = [{include = "extra_second_clone"}]

[tool.poetry.dependencies]
python = "^3.10"
poetry-core = {git = "https://github.com/python-poetry/poetry-core", extras = ["demo"]}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

$ poetry lock -vvv
Loading configuration file /home/neersighted/.config/pypoetry/config.toml
Loading configuration file /home/neersighted/.config/pypoetry/auth.toml
Trying to detect current active python executable as specified in the config.
Found: /home/neersighted/.local/share/pyenv/versions/3.11.0/bin/python
Virtualenv extra-second-clone-IgA_y9qo-py3.11 already exists.
Using virtualenv: /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: extra-second-clone is 0.1.0
   1: derived: extra-second-clone
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /python-poetry/poetry-core/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/python-poetry/poetry-core at 'main' to /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11/src/poetry-core
   1: fact: extra-second-clone depends on poetry-core[demo] (1.3.2)
   1: selecting extra-second-clone (0.1.0)
   1: derived: poetry-core[demo] (1.3.2) @ git+https://github.com/python-poetry/poetry-core@main
   1: fact: poetry-core[demo] (1.3.2) depends on poetry-core (1.3.2)
   1: selecting poetry-core[demo] (1.3.2 b0b1823)
   1: derived: poetry-core (1.3.2) @ git+https://github.com/python-poetry/poetry-core@main
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /python-poetry/poetry-core/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/python-poetry/poetry-core at 'main' to /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11/src/poetry-core
   1: selecting poetry-core (1.3.2 b0b1823)
   1: Version solving took 1.513 seconds.
   1: Tried 1 solutions.

Writing lock file

Finally, dropping the use of extras will also succeed:

[tool.poetry]
name = "extra-second-clone"
version = "0.1.0"
description = ""
authors = ["Bjorn Neergaard <bjorn@neersighted.com>"]
readme = "README.md"
packages = [{include = "extra_second_clone"}]

[tool.poetry.dependencies]
python = "^3.10"
poetry-core = {git = "https://github.com/python-poetry/poetry-core"}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

$ poetry lock -vvv
Loading configuration file /home/neersighted/.config/pypoetry/config.toml
Loading configuration file /home/neersighted/.config/pypoetry/auth.toml
Trying to detect current active python executable as specified in the config.
Found: /home/neersighted/.local/share/pyenv/versions/3.11.0/bin/python
Virtualenv extra-second-clone-IgA_y9qo-py3.11 already exists.
Using virtualenv: /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: extra-second-clone is 0.1.0
   1: derived: extra-second-clone
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /python-poetry/poetry-core/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/python-poetry/poetry-core at 'HEAD' to /home/neersighted/.cache/pypoetry/virtualenvs/extra-second-clone-IgA_y9qo-py3.11/src/poetry-core
   1: fact: extra-second-clone depends on poetry-core (1.3.2)
   1: selecting extra-second-clone (0.1.0)
   1: derived: poetry-core (1.3.2) @ git+https://github.com/python-poetry/poetry-core
   1: selecting poetry-core (1.3.2 b0b1823)
   1: Version solving took 0.945 seconds.
   1: Tried 1 solutions.

Writing lock file

The keen-eyed reader will notice that the failures occur during a second clone (?!); when extras are present, the ref HEAD cannot be resolved by Dulwich during the second clone. The second clone seems to be a bug, and eliminating it should solve the failure; however, identifying the reason the second clone with the same ref fails would be preferred so we can better understand what is going wrong.

[neersighted]

PTAL @radoering @dimbleby @abn

[dimbleby]

duplicate #6874

[neersighted]

Not really a duplicate as a PR is not an issue, and while your PR (thanks, I was having trouble finding it) addresses the symptom (and things really shouldn't be stringly typed here -- further refactoring is likely necessary), it doesn't stop the second clone.

[dimbleby]

Meh, it's the same problem with (presumably) the same solution, I'm ok with calling it a duplicate. Whatever.

Anyway as I said in that one, I don't really understand properly what's going on. Feel free to fix it better!

[robtaylor]

I just hit the same issue..