Test: Add way to discover and filter testsets by structure

[IanButterworth]

This is a proof of concept for testset filtering based on user specified strings or regexes.
Includes support for nested testsets, enabling the parents and children of matching testsets.

Alternative to #33672

The way it does this is by:

• Pkg.test first running the discovery phase of testsets and their structure, which is based on JuliaSyntax to avoid running test code at that stage
• Then applying a filter based on user specs, recording testset IDs.
• Then when running testsets use the same ID to only run enabled testsets.

Pkg part JuliaLang/Pkg.jl#4379

i.e. testing this testset https://github.com/JuliaLang/Pkg.jl/blob/482399a51bc8bea0c58cb8722fd7ddf7637aff77/test/misc.jl#L12

$ julia -e 'using Pkg; Pkg.test("Pkg", testsets=["hashing"])'

or could have used a regex match

$ julia -e 'using Pkg; Pkg.test("Pkg", testsets=[r"hashin*"])'

Developed with Claude.

[IanButterworth]

Two observations..

1. the JuliaSyntax approach can't currently see through runtime things like this.. https://github.com/JuliaLang/julia/blob/7382b7a926ee548bd2ae2b1b4a1bf17edffb153b/stdlib/Dates/test/runtests.jl#L7C1-L9C4

% ./usr/bin/julia -e 'using Pkg; Pkg.test("Dates", testsets = ["dayofweek"])'                               
     Testing Dates
...
     Testing Running tests...
Testset filtering active: 0 testsets enabled from patterns Union{Regex, String}["dayofweek"]
Enabled testsets:
     Testing Dates tests passed

2. It should probably error if no testset matches

[Seelengrab]

How does this handle custom <: AbstractTestset, if at all?

[Seelengrab]

Does this mean that adding some newlines changes this ID? Or adding new lines to a macro wrapping @testset?

[IanButterworth]

The ID just needs to be stable between the discovery and runtime. It doesn't need to be stable otherwise.

[Seelengrab]

Is there any chance for this ending up in some cache that could cause these to go out of sync, e.g. in a language server?

[Keno]

I'm suspicious of doing this syntax based. I'd rather have something that e.g. redefines the testset macro in the current scope and then runs the tests. Syntax-based things tend to have ill-defined semantics.

[IanButterworth]

I think I agree. The amount of code required to make this syntax approach work on the stdlibs alone is huge, and likely to be incomplete and brittle.

Redefining the macro for a standard run discovery phase, or a switch to disable code within the macro is the first approach I tried and failed at, but perhaps this PoC is enough to guide that to success.

[IanButterworth]

I've extended this to support Pkg.test("Dates", testsets = ["dayofweek"]) for completeness, but will investigate alternatives as mentioned above