difftest: nextest support and speedups #334

Firestar99 · 2025-07-14T17:10:37Z

Some difftest refactors of things I just want to have:

switch to mimic-libtest from tester
add profile-based test filtering via nextest, as before:
- cargo nextest run will ignore difftests
- cargo difftest will only run difftests
output is now in tests/difftests/tests/target/difftest so you can easily inspect the output instead of having to search your tmp dir
moved runner to new runner crate, bin just wraps it as a "test without harness"
- split was needed to have both tests within the runner and exposing it as a test runner
moved lib into tests/difftests/tests workspace
- prevents "Updating crates.io index" on every single test, causing massive slowdowns
- created types crate with shared mod config between lib and runner
- we no longer compile rustc_codegen_spirv, wgpu, ash etc. in both workspaces, now only in difftest workspace

Speedups

master reference
rerunning difftest without changes:
- Linux: 45s -> 6s
- Windows: 195s -> 80s (whatever is going on there)
CI difftest times:
- Linux: 12m -> 8m
- Windows: 22m -> 16m
total CI time: 23m -> 17m

Firestar99 · 2025-07-15T11:27:52Z

Even if you specify nextest filters that exclude a certain package, nextest will still build it, just not run it to discover any potential tests. I'll have to add back in the manual package excludes in CI to get back to previous CI speeds.

feature request: nextest-rs/nextest#154

Firestar99 · 2025-07-15T22:03:06Z

@LegNeato this one is ready to review, apart from the macOS difftest failing. Could you do me a favor and debug that one on your mac? It's only difftests::arch::workgroup_memory failing with I/O error: No such file or directory (os error 2) when running the ash variant.
(especially since github is taking forever to launch CI again...)

LegNeato · 2025-07-16T11:47:19Z

Pushed a fix. The problem was you were making the output directory and coming up with temp_output_path, but you weren't creating temp_output_path. Preivously the contract was the harness creates the file and passes the path to the test. The tests expected the file to be created by the harness so they could just blindly open and write and the output Differs expected the file to exist so they could just read.

LegNeato · 2025-07-16T11:48:28Z

.github/workflows/ci.yaml

        run: cargo fetch --locked --manifest-path=tests/difftests/tests/Cargo.toml --target $TARGET
-      - name: test difftest
-        run: cargo test -p "difftest*" --release --no-default-features --features "use-installed-tools"
+      - name: test difftest-runner


My main concern here is we are making CI smarter when best practice is to make it as dumb as possible.

I'm still a little unsure what to do about precompiling the tests, which is not just an CI issue but also when building from your own machine. Currently, it'll look like the tests are just stuck eating CPU for minutes, apart from the occasional slow warning from nextest, as all tests are blocked on the first test holding the target folder lock and compiling all dependencies. So at least for CI, instead of windows taking 7-8min of "nothing happening", we get cargo's compilation progress reporting.

We could move precompiling tests to the listing stage, but that would make nextest list take forever on first execution. Not even sure if we can even print compilation progress, as when listing tests we must only write a list of tests to stdout and nothing else, otherwise nextest will error out. Though I haven't tested if piping through stderr would work.

Maybe we could alias cargo difftest to be proceeded by a test compile before running nextest?

Isn't that basically what we had? It ran cargo build --release in the tests workspace regardless of what was running. The thinking was that most of the deps would be shared, but I guess I soon as I added ash vs wgpu that went out the window.

Crates in a workspace will compile all their dependencies into the target dir of their workspace, regardless of where they're from. So on master, the difftest lib crate would be compiled once in the difftest workspace's target and a second time in the root workspace's target. Now only types is shared like that, and a lot lighter since it's just plain types and no wgpu or ash.

tests/difftests/bin/src/main.rs

tests/difftests/runner/src/lib.rs

tests/difftests/tests/rust-toolchain.toml

LegNeato · 2025-07-16T11:55:02Z

nextest list compiles things like wgpu, is that expected?

LegNeato · 2025-07-16T11:56:30Z

nextest list also doesn't show any compiletests or difftests and nextest test doesn't run them...I would expect it to run all of them by default so we can tell people "install nextest, run nextest test before putting up a PR". But I guess this PR is just replacing the impl for cargo difftest and that can wait.

BTW, running cargo test from the root now appears to run difftests...

Firestar99 · 2025-07-16T21:27:56Z

nextest has a -P profile option to select a profile from the folllowing ones I configured:

rust-gpu/.config/nextest.toml

Lines 1 to 10 in 506469f

    
           [profile.default] 
        
           default-filter = '!package(difftest*) & !package(compiletest*) & !package(example-runner-*)' 
        
           fail-fast = false 
        
           [profile.difftests] 
        
           default-filter = 'package(difftests)' 
        
           slow-timeout = "2m" 
        
           [profile.difftest-runner] 
        
           default-filter = 'package(difftest-runner) | package(difftest-types)'

And cargo difftest just points nextest to the difftest profile:

rust-gpu/.cargo/config.toml

Line 3 in 506469f

difftest = "nextest run --release -P difftests -p difftests"

The only problem is that nextest compiles all crates, regardless of the profile excluding the crate / package or not. So in CI and the difftest alias I use cargo's -p option to only select relevant packages, to compile fewer crates. See this nextest feature request.

nextest test doesn't run [difftests]

With nextest run -P difftest you can. Let me know if you'd rather have the default test everything (including compiletests as well? #318)

nextest list compiles things like wgpu, is that expected?

In that sense, yes it is expected, since nextest needs to call the test binary to figure out which tests are contained within. Although, nextest run matches previous behavior on master where example-runner-wgpu and the "old" difftest lib (now moved to the difftest workspace) would force you to compile wgpu anyway.

BTW, running cargo test from the root now appears to run difftests...

cargo test does not care for nextest profiles (obviously), not much I can do there.

Firestar99 · 2025-07-16T21:50:42Z

tests/difftests/tests/Cargo.toml


+# Cargo Windows bug: workspace dependencies of crates.io dependencies are resolved
+# incorrectly in the root workspace instead of this difftest workspace. So all
+# workspace dependencies here must be mirrored into the root workspace as well.


I still don't fully understand what's causing this, I tried creating a minimal repro repo but haven't been able to. How difftest calls cargo (--manifest-path or inherited env vars) seems to be important in the repro, as you can just cargo build in the tests/difftest/tests repo and it'll build just fine.

Firestar99 · 2025-08-18T09:41:19Z

@LegNeato what's the state on this PR?

We may want to eventually switch to ui_test anyway (see #318) and get rid of nextest again. Though I feel like this PR has a number of improvements, so we should either merge this anyway, merge the non-nextest part of it or I switch it to ui_test right away. (Though finishing graphite has priority rn)

LegNeato · 2025-08-31T15:24:22Z

I tried to switch rust-cuda to ui_test from the start and hit nothing but pain. IIRC ui_test doesn't really have directives and such built in so you have to reimplement all that yourself...compiletest has a lot more built-in.

LegNeato

We need docs that nextest is now required. I think we should also switch over compiletests so that everything is consistent.

crates/rustc_codegen_spirv/src/builder/spirv_asm.rs

LegNeato · 2025-09-22T16:45:21Z

tests/difftests/runner/src/runner.rs

+            let package_out = self.output_dir.join(&package.relative_path);
+            fs::create_dir_all(&package_out)?;
+            debug!("Writing output to '{}'", package_out.display());
+            let config_file = package_out.join("config.json");


Why are you hardcoding these names now?

output is now in tests/difftests/tests/target/difftest so you can easily inspect the output instead of having to search your tmp dir

And since each test has their own folder, why not hardcode them?

tests/difftests/runner/src/testcase.rs

… crate `types`, massive difftest speedup 36s -> 6s

…cies

Firestar99 · 2025-10-16T10:10:16Z

Rebased. Had to drop "difftest: remove glam from workspace, use reeexport via spirv-std" as #380 needs to change the features on glam

Firestar99 force-pushed the difftest_refactor branch 2 times, most recently from 1215a7c to 5b43740 Compare July 15, 2025 11:03

Firestar99 mentioned this pull request Jul 15, 2025

Add filtering for packages, not just tests nextest-rs/nextest#154

Open

Firestar99 force-pushed the difftest_refactor branch 7 times, most recently from 0677196 to a861aa2 Compare July 15, 2025 19:59

Firestar99 force-pushed the const_folding branch from d33b067 to d8ec881 Compare July 15, 2025 20:02

Firestar99 force-pushed the difftest_refactor branch 3 times, most recently from ec23891 to da0bb51 Compare July 15, 2025 21:36

Firestar99 force-pushed the difftest_refactor branch from da0bb51 to 55e2ffb Compare July 15, 2025 22:06

Firestar99 changed the base branch from const_folding to main July 15, 2025 22:06

Firestar99 force-pushed the difftest_refactor branch from 55e2ffb to 33dbe33 Compare July 15, 2025 22:17

Firestar99 added a commit that referenced this pull request Jul 16, 2025

difftest: fixup windows dependency resolution, from PR #334

6e1980b

LegNeato reviewed Jul 16, 2025

View reviewed changes

LegNeato force-pushed the difftest_refactor branch from e74b17e to d3a1cab Compare July 16, 2025 11:53

Firestar99 commented Jul 16, 2025

View reviewed changes

Firestar99 changed the title ~~difftest refactors~~ difftest: nextest support and speedups Jul 17, 2025

Firestar99 force-pushed the difftest_refactor branch 2 times, most recently from 065dc99 to 606e9ab Compare July 21, 2025 11:04

Firestar99 force-pushed the difftest_refactor branch from 6295980 to d01f0c6 Compare July 28, 2025 10:56

Firestar99 force-pushed the difftest_refactor branch from d01f0c6 to 0c2df59 Compare September 8, 2025 11:16

Firestar99 mentioned this pull request Sep 8, 2025

abi: Make glam types abi layout match CPU #380

Merged

Firestar99 force-pushed the difftest_refactor branch from 0c2df59 to 23fe024 Compare September 10, 2025 09:11

Firestar99 force-pushed the difftest_refactor branch 2 times, most recently from 1b4b2d9 to 24a9bc9 Compare September 18, 2025 14:55

LegNeato requested changes Sep 22, 2025

View reviewed changes

Firestar99 force-pushed the difftest_refactor branch 2 times, most recently from 9d02ac3 to 938c511 Compare September 23, 2025 09:20

Firestar99 and others added 14 commits October 16, 2025 11:43

difftest: use relative paths, create output directory in target

bc24f29

difftest: fix test mod paths with subdirectories

950e2e2

difftest: fix test names on windows

4c869e5

difftest: move difftest bin to runner

467d408

difftest: switch to mimic-libtest from tester

48434d6

difftest: disable logger when listing available tests

fa71c03

difftest: fix use-installed-tools feature

e7fa026

difftest: use nextest to properly filter tests, adjust ci

0583730

difftest: move lib into tests workspace, move mod config to new…

7c1fa69

… crate `types`, massive difftest speedup 36s -> 6s

difftest: compile difftest in ci before running

d1a4989

ci: do not fail-fast matrices

1262640

difftest: better io error reporting in config

0a47c9e

difftest: cargo windows bug workaround, mirror all workspace dependen…

56e3fa2

…cies

difftest: Unconditionally create output file

c8895f2

Firestar99 force-pushed the difftest_refactor branch from 938c511 to e4158b5 Compare October 16, 2025 10:03

Firestar99 added 2 commits October 16, 2025 12:05

difftest: typo fixes

908a25a

difftest: document nextest requirement

78ffc4a

Firestar99 force-pushed the difftest_refactor branch from e4158b5 to 78ffc4a Compare October 16, 2025 10:08

difftest: nextest support and speedups #334

Are you sure you want to change the base?

difftest: nextest support and speedups #334

Uh oh!

Conversation

Firestar99 commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speedups

Uh oh!

Firestar99 commented Jul 15, 2025

Uh oh!

Firestar99 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Firestar99 Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

LegNeato Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Firestar99 Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Firestar99 commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Firestar99 Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Firestar99 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LegNeato commented Aug 31, 2025

Uh oh!

LegNeato left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LegNeato Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

Firestar99 Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Firestar99 commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Firestar99 commented Jul 14, 2025 •

edited

Loading

Firestar99 commented Jul 15, 2025 •

edited

Loading

LegNeato commented Jul 16, 2025 •

edited

Loading

Firestar99 commented Jul 16, 2025 •

edited

Loading

Firestar99 Jul 16, 2025 •

edited

Loading

Firestar99 commented Aug 18, 2025 •

edited

Loading

Firestar99 commented Oct 16, 2025 •

edited

Loading