Making compiletest more maintainable: reworking directive handling

[nikomatsakis]

Metadata
Point of contact	@jieyouxu
Team(s)	bootstrap, compiler, rustdoc
Goal document	2025h1/compiletest-directive-rework

Summary

Short description of what you will do over the next 6 months.

Rework [compiletest]'s directive handling to make it more maintainable, have better UX for
compiler contributors, and fix some long-standing issues.

Tasks and status

• Discussion and moral support (bootstrap, compiler, rustdoc )
• Experimental prototype implementation[^1] (@jieyouxu)
• [compiletest] changes w/ experience from prototype (@jieyouxu)
• Standard reviews (bootstrap, compiler )
• Inside Rust blog post for project outcome (@jieyouxu)

[nikomatsakis]

This issue is intended for status updates only.

For general questions or comments, please contact the owner(s) directly.

[jieyouxu]

Update (2025-02-19):

• To make it easier to follow bootstrap's test flow going into running compiletest-managed test suites, I've added more tracing to bootstrap in bootstrap: add more tracing to compiler/std/llvm flows rust#137080.
• There are some prerequisite cleanups/improvements that I'm working on first to make it easier to read bootstrap + compiletest's codebase for reference: Breakup test.rs into more-manageable sub-modules rust#137224, [compiletest-related cleanups 3/7] Make the distinction between sources root vs test suite sources root in compiletest less confusing rust#136474, [compiletest-related cleanups 4/7] Make the distinction between root build directory vs test suite specific build directory in compiletest less confusing rust#136542
• I'm thinking for the prototype I'm going to experiment with a branch off of rust-lang/rust instead of completely separately, so I can experiment under the context of bootstrap and tests that we actually have, instead of trying to experiment with it in a complete vacuum (esp. with respect to staging and dependency licenses).
  • Onur is working on stabilize stage management for rustc tools #137215 which I will wait on to match ToolStd behavior as used by compiletest.

[jieyouxu]

Update (2025-02-27):

• Cleanups still waiting to be merged (some PRs are blocked on changes from others making this slow).

[jieyouxu]

Update (2025-03-15):

• Doing a survey pass on compiletest to make sure I have the full picture.

[jieyouxu]

Update (2025-05-01):

• Not much updates, recent compiletest changes were surrounding error annotation strictness/canonicalization and landing a new executor that doesn't depend on libtest, and I've mostly been involved in reviewing those.
• Next planned changes are first to introduce some discipline into compiletest's error handling and contributor-facing diagnostics, because configuration and directive handling currently still has a ton of random panics all over the place.