docs.rs link is missing from search results

[jturner314]

For crates without a documentation key in their Cargo.toml, crates.io is supposed to generate a Documentation link to docs.rs. However, this Documentation link is sometimes missing on search results.

Steps to reproduce:

1. Open a fresh browser instance. (For example, I'm using chromium --user-data-dir="$(mktemp -d)" on Arch Linux.)
2. Navigate to https://crates.io/search?q=ndarray-npy
3. Observe that there is no Documentation link for the ndarray-npy crate. (It should be there, but it is missing. That's what this issue is about.)
4. Click on the ndarray-npy crate link (URL: https://crates.io/crates/ndarray-npy).
5. Click the "back" button on your browser to navigate back to the previous page.
6. Observe that the Documentation link has now appeared.

I'm observing this behavior for crates that do not include a documentation key in their Cargo.toml. Currently, examples of this include the ndarray-npy, py_literal, aho-corasick, unreachable, and void crates, among many others.

[jtgeibel]

I think this is a general bug (not just for crates that are missing a documentation key). I think similar behavior will be seen with the following fields as well:

• keywords
• categories
• homepage
• documentation

(I'm not sure if any of these other fields show up on the results page. This may be the only missing field that is observable.)

The search results code constructs the crate data by using the minimal_encodable function, and this defaults those fields to null in the JSON response. Once you click through to the full page for the crate, then a full response is received and cached for the crate. This is why the link shows up after navigating back to the results.

[carols10cents]

When we added auto-linking to docs.rs in this PR, we discussed adding this to the search results page as well. However, we make a request to docs.rs to see if the docs were built successfully before adding the link to the page. We concluded that making a large number of requests to docs.rs wouldn't be a great user experience.

Some other thoughts:

• This is even more of a concern since we just increased the default number of results per page from 10 to 50.
• Now that we have a background job framework, after publishing a crate, we could enqueue a job that polls docs.rs until getting a build status or hitting some timeout, then storing the URL in that version's documentation field if the build_status is true and stopping polling if the build_status is false. Oops, just checked-- documentation is still a per-crate field, not per-version, so for this to work correctly we'd have to move that field to the versions table instead. Or we could watch the feed of releases from docs.rs, which would limit the requests we make of docs.rs to a number independent of the number of crates published. I think this is likely the direction I'd prefer, but whew, this is a bigger change than I expected.
• Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

[withoutboats]

Hey, not to be a bother but I just wanted to mention that now that the auto linking to docs.rs has existed for a while, it has become very common for libraries to no longer include their own documentation key in Cargo.toml. This means it increasingly seems like most libraries don't have links to documentation on the search index.

Sucks that this is such a big change, but if someone were looking for something to hack on in crates.io it would be very appreciated!

[jyn514]

Onur mentioned in that PR thinking about adding a way to ask docs.rs about the status of many crates at once; as far as i can tell that hasn't been added but there are building blocks that wouldn't make it too bad. I don't know if it's worth adding such a thing to docs.rs if we're going to be the only consumer of it. Waiting for that one request to return and then updating the page to have a bunch of doc links or not might also be a non-optimal user experience.

I would be ok with adding an API for fetching many crates at once. I do think we should tailor the API for this use case a bit and only show the status, not the crate details, to make it cheaper. (Right now we show a bunch of metadata about the versions: https://docs.rs/crate/tokio/1.0.0/builds.json)

[jtgeibel]

In our team meeting we discussed that on the search results page it probably makes sense to just link to the crate on docs.rs without providing a version. Then docs.rs would redirect to the latest rendered version available.

This might open small windows of inconsistency, such that if a new version was just published then the search results might show version 0.4.1 but the link redirects to 0.4.0 because the new version hasn't been rendered yet. Or a brand new crate could show up in search results but have no rendered version at all on docs.rs. However, this tradeoff seems acceptable in order to consistently show a fallback link to docs.rs if no link is explicitly provided in Cargo.toml.

[jyn514]

@jtgeibel what if the build for the crate failed? Then crates.io will link to the docs, but they won't be there.

This will also mean you get newer versions that the one currently published, which could be very confusing if they're different major versions.

[jyn514]

This will also mean you get newer versions

Oh wait, I guess crates.io search only shows the latest version. Never mind then.

[eggyal]

Perhaps docs.rs can provide a hook by which crates.io gets notified whenever a new doc build is completed? Then crates.io can update its metadata for that crate without having to make requests of docs.rs.

[jyn514]

Heh, docs.rs was actually hoping to get a hook from crates.io at some point: rust-lang/docs.rs#14. I guess it wouldn't hurt to have both.