Skip to content

Make cache errors non-fatal in Planner::build #12281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 18, 2025
Merged

Make cache errors non-fatal in Planner::build #12281

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8552822
Make errors non-fatal for cache archive pointer reads in Planner
Gankra Mar 18, 2025
233a622
eliminate all other cache errors
Gankra Mar 18, 2025
fcf830e
bump cache version
Gankra Mar 18, 2025
34b6b63
fix caps
Gankra Mar 18, 2025
a56b632
remove fallibility from RequirementSatisfaction::check
Gankra Mar 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion crates/uv-cache/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,7 @@ impl CacheBucket {
match self {
// Note that when bumping this, you'll also need to bump it
// in `crates/uv/tests/it/cache_prune.rs`.
Self::SourceDistributions => "sdists-v8",
Self::SourceDistributions => "sdists-v9",
Self::FlatIndex => "flat-index-v2",
Self::Git => "git-v0",
Self::Interpreter => "interpreter-v4",
Expand Down
205 changes: 127 additions & 78 deletions crates/uv-installer/src/plan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,17 +97,20 @@ impl<'a> Planner<'a> {
[] => {}
[installed] => {
let source = RequirementSource::from(dist);
match RequirementSatisfaction::check(installed, &source)? {
RequirementSatisfaction::Mismatch => {
match RequirementSatisfaction::check(installed, &source) {
Ok(RequirementSatisfaction::Mismatch) => {
debug!("Requirement installed, but mismatched:\n Installed: {installed:?}\n Requested: {source:?}");
}
RequirementSatisfaction::Satisfied => {
Ok(RequirementSatisfaction::Satisfied) => {
debug!("Requirement already installed: {installed}");
continue;
}
RequirementSatisfaction::OutOfDate => {
Ok(RequirementSatisfaction::OutOfDate) => {
debug!("Requirement installed, but not fresh: {installed}");
}
Err(err) => {
debug!("Failed to deserialize cached requirements for: {installed} ({err})");
Copy link
Member

@charliermarsh charliermarsh Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want this to be warn!

Copy link
Member

@charliermarsh charliermarsh Mar 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this can fail for other reasons though. Should it be more discriminant or no?

Copy link
Contributor Author

@Gankra Gankra Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least in the current implementation, it actually is all CacheInfo calls that make this fallible. warn feels too aggressive for "oops the cache was invalid"?

Copy link
Member

@charliermarsh charliermarsh Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we should always know if we hit this though, it's almost always a sign of a bug. I'm honestly wondering if this should fail hard in debug?

Copy link
Member

@charliermarsh charliermarsh Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that's probably out-of-scope.

Copy link
Member

@charliermarsh charliermarsh Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this into RequirementSatisfaction::check? If it's already non-fatal for other kinds of errors, maybe it shouldn't return Result at all and we should just ignore these errors within that method.

Copy link
Contributor Author

@Gankra Gankra Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a good idea... hmm interesting SitePackages::satisfies_requirements also ?s this method and it probably shouldn't..?

Copy link
Contributor Author

@Gankra Gankra Mar 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented as new commit.

}
}
reinstalls.push(installed.clone());
}
Expand Down Expand Up @@ -180,24 +183,31 @@ impl<'a> Planner<'a> {
.entry(format!("{}.http", wheel.filename.cache_key()));

// Read the HTTP pointer.
if let Some(pointer) = HttpArchivePointer::read_from(&cache_entry)? {
let cache_info = pointer.to_cache_info();
let archive = pointer.into_archive();
if archive.satisfies(hasher.get(dist.as_ref())) {
let cached_dist = CachedDirectUrlDist {
filename: wheel.filename.clone(),
url: VerbatimParsedUrl {
parsed_url: wheel.parsed_url(),
verbatim: wheel.url.clone(),
},
hashes: archive.hashes,
cache_info,
path: cache.archive(&archive.id),
};

debug!("URL wheel requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
match HttpArchivePointer::read_from(&cache_entry) {
Ok(Some(pointer)) => {
let cache_info = pointer.to_cache_info();
let archive = pointer.into_archive();
if archive.satisfies(hasher.get(dist.as_ref())) {
let cached_dist = CachedDirectUrlDist {
filename: wheel.filename.clone(),
url: VerbatimParsedUrl {
parsed_url: wheel.parsed_url(),
verbatim: wheel.url.clone(),
},
hashes: archive.hashes,
cache_info,
path: cache.archive(&archive.id),
};

debug!("URL wheel requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}
debug!("Cached URL wheel requirement does not match expected hash policy for: {wheel}");
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached URL wheel requirement for: {wheel} ({err})");
}
}
}
Expand Down Expand Up @@ -230,28 +240,41 @@ impl<'a> Planner<'a> {
)
.entry(format!("{}.rev", wheel.filename.cache_key()));

if let Some(pointer) = LocalArchivePointer::read_from(&cache_entry)? {
let timestamp = Timestamp::from_path(&wheel.install_path)?;
if pointer.is_up_to_date(timestamp) {
let cache_info = pointer.to_cache_info();
let archive = pointer.into_archive();
if archive.satisfies(hasher.get(dist.as_ref())) {
let cached_dist = CachedDirectUrlDist {
filename: wheel.filename.clone(),
url: VerbatimParsedUrl {
parsed_url: wheel.parsed_url(),
verbatim: wheel.url.clone(),
},
hashes: archive.hashes,
cache_info,
path: cache.archive(&archive.id),
};

debug!("Path wheel requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
match LocalArchivePointer::read_from(&cache_entry) {
Ok(Some(pointer)) => {
match Timestamp::from_path(&wheel.install_path) {
Ok(timestamp) => {
if pointer.is_up_to_date(timestamp) {
let cache_info = pointer.to_cache_info();
let archive = pointer.into_archive();
if archive.satisfies(hasher.get(dist.as_ref())) {
let cached_dist = CachedDirectUrlDist {
filename: wheel.filename.clone(),
url: VerbatimParsedUrl {
parsed_url: wheel.parsed_url(),
verbatim: wheel.url.clone(),
},
hashes: archive.hashes,
cache_info,
path: cache.archive(&archive.id),
};

debug!("Path wheel requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}
debug!("Cached path wheel requirement does not match expected hash policy for: {wheel}");
}
}
Err(err) => {
debug!("failed to get timestamp for wheel {wheel} ({err})");
}
}
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached path wheel requirement for: {wheel} ({err})");
}
}
}
Dist::Source(SourceDist::Registry(sdist)) => {
Expand Down Expand Up @@ -281,19 +304,27 @@ impl<'a> Planner<'a> {
Dist::Source(SourceDist::DirectUrl(sdist)) => {
// Find the most-compatible wheel from the cache, since we don't know
// the filename in advance.
if let Some(wheel) = built_index.url(sdist)? {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_url_dist(sdist);
debug!("URL source requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}
match built_index.url(sdist) {
Ok(Some(wheel)) => {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_url_dist(sdist);
debug!("URL source requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}

warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
}
Ok(None) => {}
Err(err) => {
debug!(
"Failed to deserialize cached wheel filename for: {sdist} ({err})"
);
}
}
}
Dist::Source(SourceDist::Git(sdist)) => {
Expand Down Expand Up @@ -322,19 +353,27 @@ impl<'a> Planner<'a> {

// Find the most-compatible wheel from the cache, since we don't know
// the filename in advance.
if let Some(wheel) = built_index.path(sdist)? {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_path_dist(sdist);
debug!("Path source requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}
match built_index.path(sdist) {
Ok(Some(wheel)) => {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_path_dist(sdist);
debug!("Path source requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}

warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
}
Ok(None) => {}
Err(err) => {
debug!(
"Failed to deserialize cached wheel filename for: {sdist} ({err})"
);
}
}
}
Dist::Source(SourceDist::Directory(sdist)) => {
Expand All @@ -345,19 +384,29 @@ impl<'a> Planner<'a> {

// Find the most-compatible wheel from the cache, since we don't know
// the filename in advance.
if let Some(wheel) = built_index.directory(sdist)? {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_directory_dist(sdist);
debug!("Directory source requirement already cached: {cached_dist}");
cached.push(CachedDist::Url(cached_dist));
continue;
}
match built_index.directory(sdist) {
Ok(Some(wheel)) => {
if wheel.filename.name == sdist.name {
let cached_dist = wheel.into_directory_dist(sdist);
debug!(
"Directory source requirement already cached: {cached_dist}"
);
cached.push(CachedDist::Url(cached_dist));
continue;
}

warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
warn!(
"Cached wheel filename does not match requested distribution for: `{}` (found: `{}`)",
sdist,
wheel.filename
);
}
Ok(None) => {}
Err(err) => {
debug!(
"Failed to deserialize cached wheel filename for: {sdist} ({err})"
);
}
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion crates/uv/tests/it/cache_prune.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ fn prune_stale_revision() -> Result<()> {
----- stderr -----
DEBUG uv [VERSION] ([COMMIT] DATE)
Pruning cache at: [CACHE_DIR]/
DEBUG Removing dangling source revision: [CACHE_DIR]/sdists-v8/[ENTRY]
DEBUG Removing dangling source revision: [CACHE_DIR]/sdists-v9/[ENTRY]
DEBUG Removing dangling cache archive: [CACHE_DIR]/archive-v0/[ENTRY]
Removed [N] files ([SIZE])
"###);
Expand Down
Loading