Provide index to pip for installation of sdist build dependencies

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10, Version 1909
• Poetry version: 1.1.3
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/ajacquah/4b6cab23073fb21c03eaa436b911f89d

Issue

Pip as invoked by poetry installs build depdencies from pypi.org instead of custom repository. And since I am running this behind a firewall with no proxy settings this fails. My understanding was that poetry would use default repository for all package actions. Can you please take a look?

To reproduce:

1. Create new project. $poetry new --src test-pkg
2. Add a private default source. This source is a mirror of pypi.org + internal packages.
3. From limited connectivity environment, try to add black dev dependency: poetry add black -D

Result stack trace:

Command C:\Source\test-pkg\.venv\Scripts\pip.exe install --no-deps file:///C:/Users/ajacquah/AppData/Local/pypoetry/Cache/artifacts/8b/b9/b7/d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727/black-20.8b1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\ajacquah\appdata\local\pypoetry\cache\artifacts\8b\b9\b7\d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727\black-20.8b1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Source\rails-test\.venv\Scripts\python.exe' 'C:\Source\rails-test\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
         cwd: None
    Complete output (7 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CB670>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBAF0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBC40>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4CBEE0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000278EC4A3BB0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
    ERROR: No matching distribution found for setuptools>=41.0
    ----------------------------------------
  ERROR: Command errored out with exit status 1: 'C:\Source\test-pkg\.venv\Scripts\python.exe' 'C:\Source\test-pkg\.venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-x0saifh5\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.


  at c:\python38\lib\site-packages\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):

Thanks for your help!

[abn]

@ajacquah can you provide the output of the following please?

poetry add black -D -vvv

# if black is already in the pyproject.toml you can do this too
poetry lock -vvv

@abn here's the output

Microsoft Windows [Version 10.0.18363.1082]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Source\test-pkg>poetry add black -D -vvv
Using virtualenv: C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8
prod-pypi: 0 packages found for black *
prod-pypi: 1 packages found for black *
Using version ^20.8b1 for black

Updating dependencies
Resolving dependencies...
   1: fact: test-pkg is 0.1.0
   1: derived: test-pkg
   1: fact: test-pkg depends on pytest (^5.2)
   1: fact: test-pkg depends on black (^20.8b1)
   1: fact: test-pkg depends on pytest (^5.2)
   1: fact: test-pkg depends on black (^20.8b1)
   1: selecting test-pkg (0.1.0)
   1: derived: black (^20.8b1)
   1: derived: pytest (^5.2)
prod-pypi: 1 packages found for black >=20.8b1,<21.0
   1: fact: black (20.8b1) depends on click (>=7.1.2)
   1: fact: black (20.8b1) depends on appdirs (*)
   1: fact: black (20.8b1) depends on toml (>=0.10.1)
   1: fact: black (20.8b1) depends on typed-ast (>=1.4.0)
   1: fact: black (20.8b1) depends on regex (>=2020.1.8)
   1: fact: black (20.8b1) depends on pathspec (>=0.6,<1)
   1: fact: black (20.8b1) depends on typing_extensions (>=3.7.4)
   1: fact: black (20.8b1) depends on mypy_extensions (>=0.4.3)
   1: selecting black (20.8b1)
   1: derived: mypy_extensions (>=0.4.3)
   1: derived: typing_extensions (>=3.7.4)
   1: derived: pathspec (>=0.6,<1)
   1: derived: regex (>=2020.1.8)
   1: derived: typed-ast (>=1.4.0)
   1: derived: toml (>=0.10.1)
   1: derived: appdirs (*)
   1: derived: click (>=7.1.2)
   1: fact: pytest (5.2.0) depends on py (>=1.5.0)
   1: fact: pytest (5.2.0) depends on packaging (*)
   1: fact: pytest (5.2.0) depends on attrs (>=17.4.0)
   1: fact: pytest (5.2.0) depends on more-itertools (>=4.0.0)
   1: fact: pytest (5.2.0) depends on atomicwrites (>=1.0)
   1: fact: pytest (5.2.0) depends on pluggy (>=0.12,<1.0)
   1: fact: pytest (5.2.0) depends on wcwidth (*)
   1: fact: pytest (5.2.0) depends on colorama (*)
   1: selecting pytest (5.2.0)
   1: derived: colorama (*)
   1: derived: wcwidth (*)
   1: derived: pluggy (>=0.12,<1.0)
   1: derived: atomicwrites (>=1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: packaging (*)
   1: derived: py (>=1.5.0)
   1: selecting mypy-extensions (0.4.3)
   1: selecting typing-extensions (3.7.4.3)
   1: selecting pathspec (0.8.0)
   1: selecting regex (2020.9.27)
   1: selecting typed-ast (1.4.1)
   1: selecting toml (0.10.1)
   1: selecting appdirs (1.4.4)
   1: selecting click (7.1.2)
   1: selecting colorama (0.4.4)
   1: selecting wcwidth (0.2.5)
   1: selecting pluggy (0.13.1)
   1: selecting atomicwrites (1.4.0)
   1: selecting more-itertools (8.5.0)
   1: selecting attrs (20.2.0)
   1: fact: packaging (20.4) depends on pyparsing (>=2.0.2)
   1: fact: packaging (20.4) depends on six (*)
   1: selecting packaging (20.4)
   1: derived: six (*)
   1: derived: pyparsing (>=2.0.2)
   1: selecting py (1.9.0)
   1: selecting six (1.15.0)
   1: selecting pyparsing (2.4.7)
   1: Version solving took 0.200 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 19 skipped

  • Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing appdirs (1.4.4): Skipped for the following reason: Already installed
  • Installing atomicwrites (1.4.0): Skipped for the following reason: Already installed
  • Installing attrs (20.2.0): Skipped for the following reason: Already installed
  • Installing click (7.1.2): Skipped for the following reason: Already installed
  • Installing more-itertools (8.5.0): Skipped for the following reason: Already installed
  • Installing colorama (0.4.4): Skipped for the following reason: Already installed
  • Installing packaging (20.4): Skipped for the following reason: Already installed
  • Installing pluggy (0.13.1): Skipped for the following reason: Already installed
  • Installing mypy-extensions (0.4.3): Skipped for the following reason: Already installed
  • Installing pathspec (0.8.0): Skipped for the following reason: Already installed
  • Installing regex (2020.9.27): Skipped for the following reason: Already installed
  • Installing toml (0.10.1): Skipped for the following reason: Already installed
  • Installing py (1.9.0): Skipped for the following reason: Already installed
  • Installing typed-ast (1.4.1): Skipped for the following reason: Already installed
  • Installing typing-extensions (3.7.4.3): Skipped for the following reason: Already installed
  • Installing wcwidth (0.2.5): Skipped for the following reason: Already installed
  • Installing black (20.8b1)
  • Installing pytest (5.2.0): Skipped for the following reason: Already installed

  Stack trace:

  7  c:\python38\lib\site-packages\poetry\installation\executor.py:199 in _execute_operation
      197│
      198│             try:
    → 199│                 result = self._do_execute_operation(operation)
      200│             except EnvCommandError as e:
      201│                 if e.e.returncode == -2:

  6  c:\python38\lib\site-packages\poetry\installation\executor.py:273 in _do_execute_operation
      271│             return 0
      272│
    → 273│         result = getattr(self, "_execute_{}".format(method))(operation)
      274│
      275│         if result != 0:

  5  c:\python38\lib\site-packages\poetry\installation\executor.py:408 in _execute_install
      406│
      407│     def _execute_install(self, operation):  # type: (Install) -> None
    → 408│         return self._install(operation)
      409│
      410│     def _execute_update(self, operation):  # type: (Update) -> None

  4  c:\python38\lib\site-packages\poetry\installation\executor.py:446 in _install
      444│             args.insert(2, "-U")
      445│
    → 446│         return self.run_pip(*args)
      447│
      448│     def _update(self, operation):

  3  c:\python38\lib\site-packages\poetry\installation\executor.py:297 in run_pip
      295│     def run_pip(self, *args, **kwargs):  # type: (...) -> int
      296│         try:
    → 297│             self._env.run_pip(*args, **kwargs)
      298│         except EnvCommandError as e:
      299│             output = decode(e.e.output)

  2  c:\python38\lib\site-packages\poetry\utils\env.py:916 in run_pip
       914│         pip = self.get_pip_command()
       915│         cmd = pip + list(args)
    →  916│         return self._run(cmd, **kwargs)
       917│
       918│     def _run(self, cmd, **kwargs):

  1  c:\python38\lib\site-packages\poetry\utils\env.py:1189 in _run
      1187│             self.unset_env("__PYVENV_LAUNCHER__")
      1188│
    → 1189│             return super(VirtualEnv, self)._run(cmd, **kwargs)
      1190│
      1191│     def execute(self, bin, *args, **kwargs):

  EnvCommandError

  Command C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\pip.exe install --no-deps file:///C:/Users/ajacquah/AppData/Local/pypoetry/Cache/artifacts/8b/b9/b7/d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727/black-20.8b1.tar.gz errored with the following return code 1, and output:
  Processing c:\users\ajacquah\appdata\local\pypoetry\cache\artifacts\8b\b9\b7\d4ec53ca89b288b6be53361d96da13d397c1dfaf70aa7b48b127e6c727\black-20.8b1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: still running...
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\python.exe' 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-ykmktx6r\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
         cwd: None
    Complete output (7 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E6A0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E370>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E250>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E130>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x00000186F282E0A0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/
    ERROR: Could not find a version that satisfies the requirement setuptools>=41.0 (from versions: none)
    ERROR: No matching distribution found for setuptools>=41.0
    ----------------------------------------
  ERROR: Command errored out with exit status 1: 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\Scripts\python.exe' 'C:\Users\ajacquah\AppData\Local\pypoetry\Cache\virtualenvs\test-pkg-8YGbIfVO-py3.8\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\ajacquah\AppData\Local\Temp\pip-build-env-ykmktx6r\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.


  at c:\python38\lib\site-packages\poetry\utils\env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│
       950│         return decode(output)
       951│
       952│     def execute(self, bin, *args, **kwargs):


Failed to add packages, reverting the pyproject.toml file to its original content.

[abn]

@ajacquah the issue here is that the isolated build environment required to build balck from sdist is not able to be setup as it cannot download setuptools.

Poetry when building wheels from sdist for dependencies (not the project itself) will handover build to pip via pip install /path/to/sdist. This will in turn trigger a PEP 517 isolated build within pip. During this process, it needs to get the dependencies required for the build. This step is handled by pip and not by poetry in this case.

As for why the sdist is being used, you will have to check if proxy has the wheels available. If not, that could explain why an sdist is downloaded and used.

Eventually we would want to consume this part of the pipe too, and eventually setup the build environment ourselves. Once that happens this should, and a lot of other annoyances, go away.

@abn - Thanks for explaining, that makes sense. And glad to hear this part will be consumed by poetry in the future.

I believe black and possibly other projects only makes sdist releases. I will look into this and confirm.

But given that we have the option to set a private repository as the default one, shouldn't this repo url be set via the --index-url for pip when handing over the build? I think that would address this issue in the meantime, although I'm not sure how it would impact other parts of poetry or other package install scenarios.

[abn]

At initial glance, I reckon your proposal is reasonable and should not impact other aspects as only assist and path cases require the isolated build environment.

Can you maybe verify that doing so manually works in your environment?

Thanks @abn . Yes I can verify this works:

C:\Users\ajacquah>pip install black --index-url https://artifactory.com/artifactory/api/pypi/prod-pypi/simple --force-reinstall --no-cache
Looking in indexes: https://artifactory.com/artifactory/api/pypi/prod-pypi/simple
Collecting black
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/black/20.8b1/black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 1.7 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting click>=7.1.2
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/click/7.1.2/click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 5.8 MB/s
Collecting typed-ast>=1.4.0
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/typed-ast/1.4.1/typed_ast-1.4.1-cp38-cp38-win_amd64.whl (1
58 kB)
     |████████████████████████████████| 158 kB 2.2 MB/s
Collecting typing-extensions>=3.7.4
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/typing-extensions/3.7.4.3/typing_extensions-3.7.4.3-py3-no
ne-any.whl (22 kB)
Collecting toml>=0.10.1
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/toml/0.10.1/toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting mypy-extensions>=0.4.3
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/mypy-extensions/0.4.3/mypy_extensions-0.4.3-py2.py3-none-a
ny.whl (4.5 kB)
Collecting appdirs
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/appdirs/1.4.4/appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pathspec<1,>=0.6
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/pathspec/0.8.0/pathspec-0.8.0-py2.py3-none-any.whl (28 kB)

Collecting regex>=2020.1.8
  Downloading https://artifactory.com/artifactory/api/pypi/prod-pypi/regex/2020.9.27/regex-2020.9.27-cp38-cp38-win_amd64.whl (2
64 kB)
     |████████████████████████████████| 264 kB 6.4 MB/s
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-20.8b1-py3-none-any.whl size=124193 sha256=d407d2a24086842fd09d3899c5f5b39a000b7fec210843d3c01dc874cc92f5ea
  Stored in directory: C:\Users\ajacquah\AppData\Local\Temp\pip-ephem-wheel-cache-f96ou92x\wheels\fb\85\63\b3594c26bfd8e8475cf00e6c8eee29851a9ce9243a71486baa

Successfully built black
Installing collected packages: click, typed-ast, typing-extensions, toml, mypy-extensions, appdirs, pathspec, regex, black
  Attempting uninstall: click
    Found existing installation: click 7.1.2
    Uninstalling click-7.1.2:
      Successfully uninstalled click-7.1.2
  Attempting uninstall: typed-ast
    Found existing installation: typed-ast 1.4.1
    Uninstalling typed-ast-1.4.1:
      Successfully uninstalled typed-ast-1.4.1
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 3.7.4.3
    Uninstalling typing-extensions-3.7.4.3:
      Successfully uninstalled typing-extensions-3.7.4.3
  Attempting uninstall: toml
    Found existing installation: toml 0.10.1
    Uninstalling toml-0.10.1:
      Successfully uninstalled toml-0.10.1
  Attempting uninstall: mypy-extensions
    Found existing installation: mypy-extensions 0.4.3
    Uninstalling mypy-extensions-0.4.3:
      Successfully uninstalled mypy-extensions-0.4.3
  Attempting uninstall: appdirs
    Found existing installation: appdirs 1.4.4
    Uninstalling appdirs-1.4.4:
      Successfully uninstalled appdirs-1.4.4
  Attempting uninstall: pathspec
    Found existing installation: pathspec 0.8.0
    Uninstalling pathspec-0.8.0:
      Successfully uninstalled pathspec-0.8.0
  Attempting uninstall: regex
    Found existing installation: regex 2020.9.27
    Uninstalling regex-2020.9.27:
      Successfully uninstalled regex-2020.9.27
  Attempting uninstall: black
    Found existing installation: black 20.8b1
    Uninstalling black-20.8b1:
      Successfully uninstalled black-20.8b1
Successfully installed appdirs-1.4.4 black-20.8b1 click-7.1.2 mypy-extensions-0.4.3 pathspec-0.8.0 regex-2020.9.27 toml-0.10.1 typed-ast-1.4.1 typing-extensi
ons-3.7.4.3

C:\Users\ajacquah>

[qiuwei]

It seems this is already fixed in 1.1.4 release.

@qiuwei - checked and this is still an issue in 1.1.4 release.

[Woklex]

Faced a similar problem, but this problem is not present on version 1.0.10.