Skip to content

Conversation

@KristofferC
Copy link
Member

@KristofferC KristofferC commented Oct 26, 2025

instead, do all operations on top of compressed data (VersionRange -> data) instead of uncompressed (VersionNumber -> data).

This avoids having to materialize dictionaries with compat data for every version.

As an example:

Pkg.activate(; temp=true)
Pkg.add("Plots")
 _, _, reg_instance = first(Pkg.Registry.REGISTRY_CACHE)[end]

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
29.9 # MB

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
66.6 # MB

(where I am removing the size of storing the raw data of the registry itself).

and


julia> Pkg.rm(["Plots"]); empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"])

# PR
  1.320097 seconds (8.84 M allocations: 587.032 MiB, 11.50% gc time)

# master
  1.397698 seconds (9.65 M allocations: 718.492 MiB, 13.56% gc time)

@github-project-automation github-project-automation bot moved this to New in Pkg.jl Oct 26, 2025
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch 5 times, most recently from e190b0a to 3e23b37 Compare October 28, 2025 18:33
@KristofferC
Copy link
Member Author

Test seems to pass now and updated figures for

try Pkg.rm(["Plots"]; io=devnull) catch end; empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"]; io=devnull)

are

# master
  1.117075 seconds (8.07 M allocations: 644.749 MiB, 1 lock conflict)

# PR
  0.800553 seconds (6.86 M allocations: 466.209 MiB)

and specifically the deps_graph step where we gather information from the registry is

#master
  collect deps
  0.129429 seconds (2.61 M allocations: 290.017 MiB)

# PR
  collect deps
  0.063639 seconds (1.43 M allocations: 115.601 MiB)

I still need to take a proper pass through this because there is honestly quite a bit of AI slop in here, but I just wanted a finished implementation so I could look at the numbers and see it was what I thought it would be.

instead, do all operations on top of compressed data (`VersionRange` -> data) instead of uncompressed (`VersionNumber` -> data).

This avoids having to materialize dictionaries with compat data for every version.
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch from 290a468 to 3cfefbc Compare October 28, 2025 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

1 participant