Skip to content

Make metadata deserialization failures non-fatal in the cache #11105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 30, 2025
Merged

Make metadata deserialization failures non-fatal in the cache #11105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion crates/uv-cache/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -793,7 +793,7 @@ impl CacheBucket {
match self {
// Note that when bumping this, you'll also need to bump it
// in crates/uv/tests/cache_prune.rs.
Self::SourceDistributions => "sdists-v6",
Self::SourceDistributions => "sdists-v7",
Copy link
Member Author

@charliermarsh charliermarsh Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a bummer, since we're invalidating a TON of data. If we don't do this, though, then users running newer versions can hit these errors if they then check out older versions (since we'd be writing to sdists-v6, but those old versions would fail to read, since these errors are fatal right now).

It's a tradeoff. I'm undecided on what's best (bump or leave it as-is).

Copy link
Member

@zanieb zanieb Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have strong feelings here. This feels vaguely correct. Are there other cache changes we've been delaying?

Copy link
Member Author

@charliermarsh charliermarsh Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe so...

Copy link
Member Author

@charliermarsh charliermarsh Jan 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's right to bump here. Correctness and errors should be given priority.

Self::FlatIndex => "flat-index-v2",
Self::Git => "git-v0",
Self::Interpreter => "interpreter-v4",
Expand Down
153 changes: 90 additions & 63 deletions crates/uv-distribution/src/source/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -551,15 +551,21 @@ impl<'a, T: BuildContext> SourceDistributionBuilder<'a, T> {

// If the cache contains compatible metadata, return it.
let metadata_entry = cache_shard.entry(METADATA);
if let Some(metadata) = CachedMetadata::read(&metadata_entry)
.await?
.filter(|metadata| metadata.matches(source.name(), source.version()))
{
debug!("Using cached metadata for: {source}");
return Ok(ArchiveMetadata {
metadata: Metadata::from_metadata23(metadata.into()),
hashes: revision.into_hashes(),
});
match CachedMetadata::read(&metadata_entry).await {
Ok(Some(metadata)) => {
if metadata.matches(source.name(), source.version()) {
debug!("Using cached metadata for: {source}");
return Ok(ArchiveMetadata {
metadata: Metadata::from_metadata23(metadata.into()),
hashes: revision.into_hashes(),
});
}
debug!("Cached metadata does not match expected name and version for: {source}");
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached metadata for: {source} ({err})");
}
}

// Otherwise, we need a wheel.
Expand Down Expand Up @@ -882,15 +888,21 @@ impl<'a, T: BuildContext> SourceDistributionBuilder<'a, T> {

// If the cache contains compatible metadata, return it.
let metadata_entry = cache_shard.entry(METADATA);
if let Some(metadata) = CachedMetadata::read(&metadata_entry)
.await?
.filter(|metadata| metadata.matches(source.name(), source.version()))
{
debug!("Using cached metadata for: {source}");
return Ok(ArchiveMetadata {
metadata: Metadata::from_metadata23(metadata.into()),
hashes: revision.into_hashes(),
});
match CachedMetadata::read(&metadata_entry).await {
Ok(Some(metadata)) => {
if metadata.matches(source.name(), source.version()) {
debug!("Using cached metadata for: {source}");
return Ok(ArchiveMetadata {
metadata: Metadata::from_metadata23(metadata.into()),
hashes: revision.into_hashes(),
});
}
debug!("Cached metadata does not match expected name and version for: {source}");
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached metadata for: {source} ({err})");
}
}

// Otherwise, we need a source distribution.
Expand Down Expand Up @@ -1183,31 +1195,38 @@ impl<'a, T: BuildContext> SourceDistributionBuilder<'a, T> {

// If the cache contains compatible metadata, return it.
let metadata_entry = cache_shard.entry(METADATA);
if let Some(metadata) = CachedMetadata::read(&metadata_entry)
.await?
.filter(|metadata| metadata.matches(source.name(), source.version()))
{
// If necessary, mark the metadata as dynamic.
let metadata = if dynamic {
ResolutionMetadata {
dynamic: true,
..metadata.into()
match CachedMetadata::read(&metadata_entry).await {
Ok(Some(metadata)) => {
if metadata.matches(source.name(), source.version()) {
debug!("Using cached metadata for: {source}");

// If necessary, mark the metadata as dynamic.
let metadata = if dynamic {
ResolutionMetadata {
dynamic: true,
..metadata.into()
}
} else {
metadata.into()
};
return Ok(ArchiveMetadata::from(
Metadata::from_workspace(
metadata,
resource.install_path.as_ref(),
None,
self.build_context.locations(),
self.build_context.sources(),
self.build_context.bounds(),
)
.await?,
));
}
} else {
metadata.into()
};

return Ok(ArchiveMetadata::from(
Metadata::from_workspace(
metadata,
resource.install_path.as_ref(),
None,
self.build_context.locations(),
self.build_context.sources(),
self.build_context.bounds(),
)
.await?,
));
debug!("Cached metadata does not match expected name and version for: {source}");
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached metadata for: {source} ({err})");
}
}

// If the backend supports `prepare_metadata_for_build_wheel`, use it.
Expand Down Expand Up @@ -1635,27 +1654,35 @@ impl<'a, T: BuildContext> SourceDistributionBuilder<'a, T> {
.map_err(Error::CacheRead)?
.is_fresh()
{
if let Some(metadata) = CachedMetadata::read(&metadata_entry)
.await?
.filter(|metadata| metadata.matches(source.name(), source.version()))
{
let git_member = GitWorkspaceMember {
fetch_root: fetch.path(),
git_source: resource,
};

debug!("Using cached metadata for: {source}");
return Ok(ArchiveMetadata::from(
Metadata::from_workspace(
metadata.into(),
&path,
Some(&git_member),
self.build_context.locations(),
self.build_context.sources(),
self.build_context.bounds(),
)
.await?,
));
match CachedMetadata::read(&metadata_entry).await {
Ok(Some(metadata)) => {
if metadata.matches(source.name(), source.version()) {
debug!("Using cached metadata for: {source}");

let git_member = GitWorkspaceMember {
fetch_root: fetch.path(),
git_source: resource,
};
return Ok(ArchiveMetadata::from(
Metadata::from_workspace(
metadata.into(),
&path,
Some(&git_member),
self.build_context.locations(),
self.build_context.sources(),
self.build_context.bounds(),
)
.await?,
));
}
debug!(
"Cached metadata does not match expected name and version for: {source}"
);
}
Ok(None) => {}
Err(err) => {
debug!("Failed to deserialize cached metadata for: {source} ({err})");
}
}
}

Expand Down
2 changes: 1 addition & 1 deletion crates/uv/tests/it/cache_prune.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ fn prune_stale_revision() -> Result<()> {
----- stderr -----
DEBUG uv [VERSION] ([COMMIT] DATE)
Pruning cache at: [CACHE_DIR]/
DEBUG Removing dangling source revision: [CACHE_DIR]/sdists-v6/[ENTRY]
DEBUG Removing dangling source revision: [CACHE_DIR]/sdists-v7/[ENTRY]
DEBUG Removing dangling cache archive: [CACHE_DIR]/archive-v0/[ENTRY]
Removed [N] files ([SIZE])
"###);
Expand Down
Loading