Skip to content

Conversation

@konstin
Copy link
Member

@konstin konstin commented Jun 5, 2025

Surprisingly, we weren't locking during uv sync so far, so running uv sync in parallel could cause errors in filesystem races.

I've also added locks to uv add and uv remove which concurrently modify pyproject.toml. These locks only apply after we determined the interpreter, so they don't actually prevent computing the same thing twice when running uv add in parallel.

All other subcommands that I checked were already locking (with no claim to exhaustiveness)

Fixes #12751

Test Plan

I don't have CI-sized reproducer for this.

[project]
name = "debug"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
  "boto3>=1.38.30",
  "fastapi>=0.115.12",
  "numba>=0.61.2",
  "polars>=1.30.0",
  "protobuf>=6.31.1",
  "pyarrow>=20.0.0",
  "pydantic>=2.11.5",
  "requests>=2.32.3",
  "urllib3>=2.4.0",
  "scikit-learn>=1.6.1",
  "jupyter>=1.1.1",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
rm -rf .venv && parallel -n0 "uv sync -q" ::: {1..10}

Surprisingly, we weren't locking during `uv sync` so far, so running `uv sync` in parallel could cause errors in filesystem races.

I've also added locks to `uv add` and `uv remove` which concurrently modify `pyproject.toml`. These locks only apply after we determined the interpreter, so they don't actually prevent computing the same thing twice when running `uv add` in parallel.

# Test Plan

I don't have CI-sized reproducer for this.

```
[project]
name = "debug"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
  "boto3>=1.38.30",
  "fastapi>=0.115.12",
  "numba>=0.61.2",
  "polars>=1.30.0",
  "protobuf>=6.31.1",
  "pyarrow>=20.0.0",
  "pydantic>=2.11.5",
  "requests>=2.32.3",
  "urllib3>=2.4.0",
  "scikit-learn>=1.6.1",
  "jupyter>=1.1.1",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

```
rm -rf .venv && parallel -n0 "uv sync -q" ::: {1..10}
```
@konstin konstin added the bug Something isn't working label Jun 5, 2025
@leviska
Copy link

leviska commented Jun 5, 2025

Does this also lock on uv run if it needs sync?

Copy link
Contributor

@Gankra Gankra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uv version also presumably needs this

@konstin
Copy link
Member Author

konstin commented Jun 6, 2025

Tracking follow-ups in #13883

@konstin konstin merged commit bf96c60 into main Jun 6, 2025
86 checks passed
@konstin konstin deleted the konsti/lock-uv-sync branch June 6, 2025 12:16
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Jun 13, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.7.7` -> `0.7.13` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.7.13`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0713)

[Compare Source](astral-sh/uv@0.7.12...0.7.13)

##### Python

-   Add Python 3.14.0b2
-   Add Python 3.13.5
-   Fix stability of `uuid.getnode` on 3.13

See the
[`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250612)
for more details.

##### Enhancements

-   Download versions in `uv python pin` if not found ([#&#8203;13946](astral-sh/uv#13946))
-   Use TTY detection to determine if SIGINT forwarding is enabled ([#&#8203;13925](astral-sh/uv#13925))
-   Avoid fetching an exact, cached Git commit, even if it isn't locked ([#&#8203;13748](astral-sh/uv#13748))
-   Add `zstd` and `deflate` to `Accept-Encoding` ([#&#8203;13982](astral-sh/uv#13982))
-   Build binaries for riscv64  ([#&#8203;12688](astral-sh/uv#12688))

##### Bug fixes

-   Check if relative URL is valid directory before treating as index ([#&#8203;13917](astral-sh/uv#13917))
-   Ignore Python discovery errors during `uv python pin` ([#&#8203;13944](astral-sh/uv#13944))
-   Do not allow `uv add --group ... --script` ([#&#8203;13997](astral-sh/uv#13997))

##### Preview changes

-   Build backend: Support namespace packages ([#&#8203;13833](astral-sh/uv#13833))

##### Documentation

-   Add 3.14 to the supported platform reference ([#&#8203;13990](astral-sh/uv#13990))
-   Add an `llms.txt` to uv ([#&#8203;13929](astral-sh/uv#13929))
-   Add supported macOS version to the platform reference ([#&#8203;13993](astral-sh/uv#13993))
-   Update platform support reference to include Python implementation list ([#&#8203;13991](astral-sh/uv#13991))
-   Update pytorch.md ([#&#8203;13899](astral-sh/uv#13899))
-   Update the CLI help and reference to include references to the Python bin directory ([#&#8203;13978](astral-sh/uv#13978))

### [`v0.7.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0712)

[Compare Source](astral-sh/uv@0.7.11...0.7.12)

##### Enhancements

-   Add `uv python pin --rm` to remove `.python-version` pins ([#&#8203;13860](astral-sh/uv#13860))
-   Don't hint at versions removed by `excluded-newer` ([#&#8203;13884](astral-sh/uv#13884))
-   Add hint to use `tool.uv.environments` on resolution error ([#&#8203;13455](astral-sh/uv#13455))
-   Add hint to use `tool.uv.required-environments` on resolution error ([#&#8203;13575](astral-sh/uv#13575))
-   Improve `python pin` error messages ([#&#8203;13862](astral-sh/uv#13862))

##### Bug fixes

-   Lock environments during `uv sync`, `uv add` and `uv remove` to prevent race conditions ([#&#8203;13869](astral-sh/uv#13869))
-   Add `--no-editable` to `uv export` for `pylock.toml` ([#&#8203;13852](astral-sh/uv#13852))

##### Documentation

-   List `.gitignore` in project init files ([#&#8203;13855](astral-sh/uv#13855))
-   Move the pip interface documentation into the concepts section ([#&#8203;13841](astral-sh/uv#13841))
-   Remove the configuration section in favor of concepts / reference ([#&#8203;13842](astral-sh/uv#13842))
-   Update Git and GitHub Actions docs to mention `gh auth login` ([#&#8203;13850](astral-sh/uv#13850))

##### Preview

-   Fix directory glob traversal fallback preventing exclusion of all files ([#&#8203;13882](astral-sh/uv#13882))

### [`v0.7.11`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0711)

[Compare Source](astral-sh/uv@0.7.10...0.7.11)

##### Python

-   Add Python 3.14.0b1
-   Add Python 3.13.4
-   Add Python 3.12.11
-   Add Python 3.11.13
-   Add Python 3.10.18
-   Add Python 3.9.23

##### Enhancements

-   Add Pyodide support ([#&#8203;12731](astral-sh/uv#12731))
-   Better error message for version specifier with missing operator ([#&#8203;13803](astral-sh/uv#13803))

##### Bug fixes

-   Downgrade `reqwest` and `hyper-util` to resolve connection reset errors over IPv6 ([#&#8203;13835](astral-sh/uv#13835))
-   Prefer `uv`'s binary's version when checking if it's up to date ([#&#8203;13840](astral-sh/uv#13840))

##### Documentation

-   Use "terminal driver" instead of "shell" in `SIGINT` docs ([#&#8203;13787](astral-sh/uv#13787))

### [`v0.7.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0710)

[Compare Source](astral-sh/uv@0.7.9...0.7.10)

##### Enhancements

-   Add `--show-extras` to `uv tool list` ([#&#8203;13783](astral-sh/uv#13783))
-   Add dynamically generated sysconfig replacement mappings ([#&#8203;13441](astral-sh/uv#13441))
-   Add data locations to install wheel logs ([#&#8203;13797](astral-sh/uv#13797))

##### Bug fixes

-   Avoid redaction of placeholder `git` username when using SSH authentication ([#&#8203;13799](astral-sh/uv#13799))
-   Propagate credentials to files on devpi indexes ending in `/+simple` ([#&#8203;13743](astral-sh/uv#13743))
-   Restore retention of credentials for direct URLs in `uv export` ([#&#8203;13809](astral-sh/uv#13809))

### [`v0.7.9`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#079)

[Compare Source](astral-sh/uv@0.7.8...0.7.9)

##### Python

The changes reverted in [0.7.8](#&#8203;078) have been restored.

See the
[`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250529)
for more details.

##### Enhancements

-   Improve obfuscation of credentials in URLs ([#&#8203;13560](astral-sh/uv#13560))
-   Allow running non-default Python implementations via `uvx` ([#&#8203;13583](astral-sh/uv#13583))
-   Add `uvw` as alias for `uv` without console window on Windows ([#&#8203;11786](astral-sh/uv#11786))
-   Allow discovery of x86-64 managed Python builds on macOS ([#&#8203;13722](astral-sh/uv#13722))
-   Differentiate between implicit vs explicit architecture requests ([#&#8203;13723](astral-sh/uv#13723))
-   Implement ordering for Python architectures to prefer native installations ([#&#8203;13709](astral-sh/uv#13709))
-   Only show the first match per platform (and architecture) by default in `uv python list`  ([#&#8203;13721](astral-sh/uv#13721))
-   Write the path of the parent environment to an `extends-environment` key in the `pyvenv.cfg` file of an ephemeral environment ([#&#8203;13598](astral-sh/uv#13598))
-   Improve the error message when libc cannot be found, e.g., when using the distroless containers ([#&#8203;13549](astral-sh/uv#13549))

##### Performance

-   Avoid rendering info log level ([#&#8203;13642](astral-sh/uv#13642))
-   Improve performance of `uv-python` crate's manylinux submodule ([#&#8203;11131](astral-sh/uv#11131))
-   Optimize `Version` display ([#&#8203;13643](astral-sh/uv#13643))
-   Reduce number of reference-checks for `uv cache clean` ([#&#8203;13669](astral-sh/uv#13669))

##### Bug fixes

-   Avoid reinstalling dependency group members with `--all-packages` ([#&#8203;13678](astral-sh/uv#13678))
-   Don't fail direct URL hash checking with dependency metadata ([#&#8203;13736](astral-sh/uv#13736))
-   Exit early on `self update` if global `--offline` is set ([#&#8203;13663](astral-sh/uv#13663))
-   Fix cases where the uv lock is incorrectly marked as out of date ([#&#8203;13635](astral-sh/uv#13635))
-   Include pre-release versions in `uv python install --reinstall` ([#&#8203;13645](astral-sh/uv#13645))
-   Set `LC_ALL=C` for git when checking git worktree ([#&#8203;13637](astral-sh/uv#13637))
-   Avoid rejecting Windows paths for remote Python download JSON targets ([#&#8203;13625](astral-sh/uv#13625))

##### Preview

-   Add `uv add --bounds` to configure version constraints ([#&#8203;12946](astral-sh/uv#12946))

##### Documentation

-   Add documentation about Python versions to Tools concept page ([#&#8203;7673](astral-sh/uv#7673))
-   Add example of enabling Dependabot ([#&#8203;13692](astral-sh/uv#13692))
-   Fix `exclude-newer` date format for persistent configuration files ([#&#8203;13706](astral-sh/uv#13706))
-   Quote versions variables in GitLab documentation ([#&#8203;13679](astral-sh/uv#13679))
-   Update Dependabot support status ([#&#8203;13690](astral-sh/uv#13690))
-   Explicitly specify to add a new repo entry to the repos list item in the `.pre-commit-config.yaml` ([#&#8203;10243](astral-sh/uv#10243))
-   Add integration with marimo guide ([#&#8203;13691](astral-sh/uv#13691))
-   Add pronunciation to README ([#&#8203;5336](astral-sh/uv#5336))

### [`v0.7.8`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#078)

[Compare Source](astral-sh/uv@0.7.7...0.7.8)

##### Python

We are reverting most of our Python changes from `uv 0.7.6` and `uv 0.7.7` due to
a miscompilation that makes the Python interpreter behave incorrectly, resulting
in spurious type-errors involving str. This issue seems to be isolated to
x86\_64 Linux, and affected at least Python 3.12, 3.13, and 3.14.

The following changes that were introduced in those versions of uv are temporarily
being reverted while we test and deploy a proper fix for the miscompilation:

-   Add Python 3.14 on musl
-   free-threaded Python on musl
-   Add Python 3.14.0a7
-   Statically link `libpython` into the interpreter on Linux for a significant performance boost

See [the issue for details](astral-sh/uv#13610).

##### Documentation

-   Remove misleading line in pin documentation ([#&#8203;13611](astral-sh/uv#13611))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MC4yNi4xIiwidXBkYXRlZEluVmVyIjoiNDAuNTEuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiUmVub3ZhdGUgQm90Il19-->
oconnor663 added a commit that referenced this pull request Jun 20, 2025
This is very similar to the locking added for `uv sync`, `uv add`, and
`uv remove` in #13869. Improving our
(f)locking in general is tracked in
#13883.

This is a minimal change that un-breaks parallel invocations of `uv
run`. It's too easy to confuse locking as in "lockfile generation" and
locking as in "file locking to prevent race conditions", so we probably
need a broader overhaul of how these things are named. It's also too
easy to miss a callsite here, or to accidentally retain one of these
locks across `run_to_completion`, so ideally we could move this file
locking down to a lower-level function that's shared by all these
callers, possibly `pip::operations::install`. I want to get this minimal
change through testing and review before I tackle either of those.
oconnor663 added a commit that referenced this pull request Jun 20, 2025
This is very similar to the locking added for `uv sync`, `uv add`, and
`uv remove` in #13869. Improving our
(f)locking in general is tracked in
#13883.
zanieb pushed a commit that referenced this pull request Jun 21, 2025
This is very similar to the locking added for `uv sync`, `uv add`, and
`uv remove` in #13869. Improving our
(f)locking in general is tracked in
#13883.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

uv sync failure when running concurrently

4 participants