Crate metadata requests take very long on crates with a lot of features

[carloskiki]

Current Behavior

When requesting metadata through shields.io for a crate I maintain, icondata, I have to specify a specific version of the crate in order for the API request to complete.

This crate makes heavy use of feature flags which bloats the response payload of the API. A request for a specific version is fine, with a payload size of ~1MB, but when I don't specify a version i.e., when every version's metadata is included, the payload is ~20MB, which seems to make the request timeout on shields.io's side.

Expected Behavior

I know this is partly due to how shields.io works, but perhaps being able to specify the version as latest instead of a specific semver would help. An example of what it would enable use to do is request the license of the latest crate without requesting every crate's metadata.

Steps To Reproduce

GET request to https://crates.io/api/v1/crates/icondata
vs.
GET request to https://crates.io/api/v1/crates/icondata/0.1.0

Environment

• Browser: Chrome version 118.0.5993.70
• OS: Mac OS (Apple silicon m1)

Anything else?

No response

[Turbo87]

Hi @Carlosted, on behalf of the crates.io team let me tell you that we were quite surprised to see a crate with over 23.000 features... 😅

Let me first respond directly to the primary request in this issue: supporting something like /:crate_name/latest. I think this is a reasonable request, and I'm wondering if we could even generalize it to /:crate_name/:version_req so that something like /foo/^1.2.3 would redirect to /foo/1.4.0 if that was the highest matching version.

Regarding the icondata crate specifically, you will find that having that many features will surface a lot of other issues eventually. You can see for example that the index file for this crate (https://index.crates.io/ic/on/icondata) already takes up 11 MB, with only 9 versions published so far. If more versions of the crate are published, this file will quickly grow and cargo will take a loooong time resolving dependencies for any project where this crate is being used because it needs to download this big file all of the time.

We quickly discussed the situation between some of the crates.io and cargo team members today. It seems likely that we will have to introduce some kind of limit on the maximum number of features a crate can have, to keep the registry working correctly in the future.

Due to the above we would like to encourage you to think of different ways to improve the compile times without needing over 23.000 features for a single crate :)

[Turbo87]

@Carlosted 👋 there is some discussion about writing a small blog post about the situation, explaining why we introduced the limit for now. since we might want to refer to this issue and your crate we thought it might be appropriate to ask you first if you would be okay with such a blog post in general. if you would like to sign off on the final text, then we could probably ensure that as well. our plan is to write this in a blame-free and joyful way, so I wouldn't expect any negative consequences from that :)

[carloskiki]

Yes of course that's a great idea!

I'm available if you want me to approve the final text but I'm fine with not reviewing it if it makes it any easier for you :)