Local dependency and packaging

[nhuray]

Is your feature request related to a problem? Please describe.

Hi Dear community

I'm struggling to build and package an app depending on a local dependency using PDM.

So maybe it's because I'm quite new to python (long time I didn't code in that language), or because I did not understand some concepts related to PEP 582, so please excuse me if my question is obvious (or stupid).

So I want to bundle an AWS Lambda function (in Python) as a Docker image (let's name it A) following the documentation provided.

That function depends on a library (local dependency named B) which itself can bring some external dependencies (C, D, E).

So basically the dependency tree is something like:

├── A  (local)
│   └── B. (local)
│       ├── C (external)
│       ├── D (external)
│       └── E (external)

Add B as a local dependency into A project

When I add to project A the local dependency B running pdm add -e ../relative/path/to/B I have:

• an absolute path in the pyproject.toml :

  dependencies = [
      "-e file:///Absolute/path/to/B#egg=B",
  ]
  

(This looks an issue to me because the pyproject.toml is not portable anymore)

• a relative path in the pdm.lock :

name = "B"
sections = ["default"]
version = "0.0.1"
editable = true
path = "../relative/path/to/B"

• a directory __pypackages__ loooking like that:

├── __pypackages__
│   └── 3.8
│       ├── bin
│       ├── include
│       └── lib
│           ├── B.egg-link
│           ├── easy-install.pth
│           └── A.egg-link

Install A

When I run pdm install I have:

[CHANGE IN 1.5.0]: dev-dependencies are included by default and can be excluded with `--prod` option
Synchronizing working set with lock file: 0 to add, 1 to update, 0 to remove

  ✔ Update B 0.0.1 -> 0.0.1 successful
Installing the project as an editable package...
  ✔ Update A 0.0.1 -> 0.0.1 successful

🎉 All complete!

Describe the solution you'd like

What I expect when I run pdm install in the A project is to resolve the local and external dependencies then install them to the __pypackages__ directory.

When I build the Docker image like that, I just need to copy the __pypackages__/3.8/lib directory.

So how do we do that ?

[pawamoy]

The issue is that you are installing this local package in editable-mode (-e). Would it be possible for you to publish that local package to a PyPI index so you can depend on it normally?

[nhuray]

Sure but that's not exactly the philosophy when you use monorepo...

My goal is to build independent packages using local dependencies from the same monorepo without publishing those artifacts to a registry (PyPi or local python registry).

Something also discussed in the Poetry community: python-poetry/poetry#936

[pawamoy]

Maybe it would work if you specify the local package without -e, though you'd lose the editable benefits, requiring the re-run pdm install each time this local package changes.

[nhuray]

Unfortunately I tried and it's failing:

Install A 0.0.1 failed
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 184, in main
    return Core().main(args)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 145, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/core.py", line 140, in main
    f(options.project, options)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/commands/add.py", line 35, in handle
    actions.do_add(
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/actions.py", line 182, in do_add
    do_sync(project, sections=(section,), default=False)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/cli/actions.py", line 128, in do_sync
    handler.synchronize()
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/synchronizers.py", line 329, in synchronize
    handlers[self_action](self_key)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/synchronizers.py", line 148, in install_candidate
    installer.install(can)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/installers/installers.py", line 43, in install
    candidate.get_metadata(allow_all_wheels=False, raising=True)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/models/candidates.py", line 203, in get_metadata
    sdist = get_sdist(built)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/pdm/models/candidates.py", line 38, in get_sdist
    return EggInfoDistribution(egg_info) if egg_info else None
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 878, in __init__
    metadata = self._get_metadata(path)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 955, in _get_metadata
    requires = parse_requires_path(req_path)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 926, in parse_requires_path
    reqs = parse_requires_data(fp.read())
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/database.py", line 903, in parse_requires_data
    r = parse_requirement(line)
  File "/usr/local/Cellar/pdm/1.5.3/libexec/lib/python3.9/site-packages/distlib/util.py", line 193, in parse_requirement
    raise SyntaxError('Invalid URL: %s' % uri)
SyntaxError: Invalid URL: file:///Users/nicolas/Workspace/docparser/libs/B

The PDM version is 1.5.3

[frostming]

You should define your pyproject.toml as follows:

dependencies = [
    "B @ file:///${PROJECT_ROOT}/relative/path/to/B"
]

${PROJECT_ROOT} is a built-in variable you should keep as is.

[AlexAegis]

I took me a while to find this solution, I feel for one this variable should be better documented around dependency handling, and should be utilized when adding local dependencies with pdm add instead of the absolute path.

[matejetz]

This seems to not be working anymore.

dependencies = [
    "B @ file:///${PROJECT_ROOT}/../relative/path/to/B"
]

runs into [KeyError]:('B', '0.0.1', 'file:///${PROJECT_ROOT}/../relative/path/to/B', False)
with some debugging it looks like this occurs because the key tuple in the candidates dictionary has the resolved path

[frostming]

You can't refer to a package beyond the project root. Simon Matejetz ***@***.***> 于2022年4月14日周四 17:34写道：

…

This seems to not be working anymore. dependencies = [ "B @ file:///${PROJECT_ROOT}/../relative/path/to/B" ] runs into [KeyError]:('B', '0.0.1', 'file:///${PROJECT_ROOT}/../relative/path/to/B', False) with some debugging it looks like this occurs because the key tuple in the candidates dictionary has the resolved path — Reply to this email directly, view it on GitHub <#464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD4UNXT6ASWWRDRMRKJ4AXDVE7RDJANCNFSM45CKAULQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[erikreppel]

@frostming this seems like a fairly arbitrary limitation, allowing it would very much help monorepos with conflicting dependencies between apps where a single project root isnt viable. Poetry supports this with the following syntax:

[tool.poetry.dependencies]
python = "^3.9"
internalpkg = { path = "../internalpkg" }

[jdoiro3]

When I use this method, run pdm build and then try to pip install the built wheel file I get the below. Is this expected? Also, let me know if I should open a new issue for this.

Processing ./dist/project--py3-none-any.whl
Processing /${PROJECT_ROOT}/local-package
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/${PROJECT_ROOT}/local-package'

[frostming]

${PROJECT_ROOT} is there for the convenience of project management, any local paths are not allowed in the wheel metadata. You must replace it with a published package name before you package and upload it to PyPI.

[adriangb]

Could there be a new type of dependency that causes the projects to be “merged” into one for the purposes of building a distribution instead of including one as a path dependency (which like you say doesn’t work)?