RFC: how to deal with poetry 1.3's new lockfile format #14857
Description
I've spent some time on this today and now want to dump context.
- Dependabot's batch of PRs this morning all propose a big change to the way the lockfile is presented. E.g. Bump ruff from 0.0.215 to 0.0.223 #14851
- Our CI uses poetry 1.2.0, which fails to read this lockfile.
- I am guessing this is due to Update poetry requirement from <1.3.0,>=1.1.15 to >=1.1.15,<1.4.0 in /python/helpers dependabot/dependabot-core#6302 allowing the use of poetry 1.3.
- Poetry 1.3 introduces a new lockfile format: feat: lock file format 2.0 (was: store package files on the package in lockfile) python-poetry/poetry#6393
- In feat: lock file format 2.0 (was: store package files on the package in lockfile) python-poetry/poetry#6393 (comment) it was proposed that the ability to read (not write) the new format should be backported to 1.2.x, which happened here [1.2] feat: forward compatibility for lock file format 2.0 python-poetry/poetry#6608
Options:
- Stop using dependabot.
- Ask dependabot authors if there's some way to pin the version of poetry that dependabot uses.
- Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.2.2.
I assumed that option (0) was undesirable. Put differently, I contend that dependabot is valuable to us. It does a good job of summarising what's changed in dependencies, and also automatically proposes updates when security advisories are made for our dependencies.
Option (1) seemed like a relatively big ask for a fairly small scenario, so I put that to one side. EDIT: The depdendabot authors seem to be aware of this problem, see dependabot/dependabot-core#1556 (comment) and the discussion following it. In dependabot/dependabot-core#1556 (comment) they point to a poetry issue python-poetry/poetry#3316 for enforcing a minimum poetry issue, which remains open. (Note that we actually want a maximum version here).
Option (2) was my next approach. I started this process in
matrix-org/setup-python-poetry#14 and matrix-org/sytest#1326. But thinking about it, I realised that anyone who wants to edit the lockfile will have to install poetry 1.3.x. This includes some parts of CI (e.g. old deps, twisted trunk); so there's no way of being consistent; option(2) is really
2'. Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.2.2 to read it, and >= 1.3.0 to write it.
but writing this down in our docs/CI sounds like a recipe for future confusion. Instead, I propose the simpler
- Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.3 to use it. (1.2.2 will suffice for read-only operations---but we wouldn't advertise this fact.)