massive benchmark panics: thread 'main' panicked at 'couldn't read string_data file: Os { code: 2, kind: NotFound, message: "No such file or directory" }

[klensy]

A lot of benches failed, as can be seen at https://perf.rust-lang.org/status.html (at least now) with errors like:

thread 'main' panicked at 'couldn't read string_data file: Os { code: 2, kind: NotFound, message: "No such file or directory" }', analyzeme/src/profiling_data.rs:50:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: "Error reading top-level stream in file `rust_grammar-12029.mm_profdata`: Expected file format version 7 but found `8`"
thread 'main' panicked at 'failed to run summarize and summarize-9.0. Errors:
summarize: failed to run summarize in "/tmp/.tmp1Gpe86/self-profile-output" with prefix "rust_grammar-12029.mm_profdata"
summarize-9.0: failed to run summarize-9.0 in "/tmp/.tmp1Gpe86/self-profile-output" with prefix "rust_grammar-12029.mm_profdata"', collector/src/rustc-fake.rs:373:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: `rust-grammar` (lib) generated 5 warnings
error: could not compile `rust-grammar`; 5 warnings emitted

But perf run didn't have any warning about that fail in report at PR rust-lang/rust#90067 or compare page https://perf.rust-lang.org/compare.html?start=1af55d19c7a9189374d89472f97dc119659bb67e&end=42983a28ab3c70728da7a9b932b667c978dd898d, it simply silently skips failed benches.

[klensy]

2 issues actually:

1. Massive bench fail
2. No sign of problems in PR message and compare page, that allows merging possibly regressing (or improving) PRs without knowing.

[Mark-Simulacrum]

Cc #1062

@rylev any thoughts on the cause here? I can try to take a look on the collector machine in a bit

[Mark-Simulacrum]

ok, checked the collector -- our cargo install was failing because the collector had an old nightly; I'm updating that now and going to manually delete results for the two failed commits so we can rerun them.

error[E0658]: or-patterns syntax is experimental
   --> collector/src/execute.rs:887:20
    |
887 |                 @ (DeserializeStatError::ParseError { .. } | DeserializeStatError::XperfError(..)),
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #54883 <https://github.com/rust-lang/rust/issues/54883> for more information
    = help: add `#![feature(or_patterns)]` to the crate attributes to enable
error[E0277]: `[&str; 3]` is not an iterator
   --> collector/src/main.rs:326:17
    |
326 |     for name in ["summarize", "crox", "flamegraph"] {
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ borrow the array with `&` or call `.iter()` on it to iterate over it
    |
    = help: the trait `Iterator` is not implemented for `[&str; 3]`
    = note: arrays are not iterators, but slices like the following are: `&[1, 2, 3]`
    = note: required because of the requirements on the impl of `IntoIterator` for `[&str; 3]`
    = note: required by `into_iter`

[Mark-Simulacrum]

OK, should be recovering now.

[klensy]

Rerun for rust-lang/rust#89100 failed: https://perf.rust-lang.org/compare.html?start=42983a28ab3c70728da7a9b932b667c978dd898d&end=6162529a01473bbb2427fa27354cbafc3c514eee

[klensy]

Queue looks not empty https://perf.rust-lang.org/status.html

Commit Date | SHA | Reason
-- | -- | --
10/20/2021, 10:21:01 AM | 6162529a01473 | #89100
10/20/2021, 5:37:48 PM | 3d71e749a2448 | #90050

but No current collection in progress. Last one finished at 10/20/2021, 9:26:48 PM local time, 1h51m25s ago.

[Mark-Simulacrum]

Should be fixed now.

[Mark-Simulacrum]

Closing as fixed, we're tracking the bad error reporting in #1062.