Add wildcard support for UI tests

[estebank]

Some diagnostics have different output in different architectures, which makes it harder to write arch agnostic UI tests. This has cropped up multiple times. Personally I've either reworked the test or changed it to a compile-fail test, but given that we'd like to consolidate on UI tests exclusively, UI tests should have a way to avoid this problem.

[nikomatsakis]

cc @petrochenkov @cengizio @kennytm -- all of whom have shown an interest in our testing infrastructure

[nikomatsakis]

I agree, we need to fix this, but I'm not sure the best way.

I'm a bit reluctant to just add random wildcards into the output, since then you can't run the "regenerate reference" script and have it work.

One thought I had was to permit multiple references, one per architecture, but that does seem to have the same problem.

Another thought I had was to permit you to add a custom file that indicates other search-and-replace pairs specific to a particular test.

I guess it really depends on how often it comes up -- maybe it's ok to have to regenerate some reference files by hand, but it seems like a drag.

[Mark-Simulacrum]

I think maybe the right way to fix this is to have an easy way to not run UI tests on more than one platform; most UI tests don't really need this I think. That is, we're primarily testing spans, not so much the error itself. Perhaps that's not the right way to think about them, though.

[kennytm]

Instead of a custom file, I think the *.rs could specify headers for TestCx::normalize_output to teach it further normalization rules, e.g. for #42304

// normalize-stderr-x86: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g
// normalize-stderr-arm: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g
// normalize-stderr-emscripten: s/fn\(\)(>?) \(32 bits\)/fn()$1 (64 bits)/g

If regex is too heavy, we could use simple literals instead:

// normalize-stderr-x86: "fn() (32 bits)" -> "fn() (64 bits)"
// normalize-stderr-x86: "fn()> (32 bits)" -> "fn()> (64 bits)"
// normalize-stderr-arm: "fn() (32 bits)" -> "fn() (64 bits)"
// normalize-stderr-arm: "fn()> (32 bits)" -> "fn()> (64 bits)"
// normalize-stderr-emscripten: "fn() (32 bits)" -> "fn() (64 bits)"
// normalize-stderr-emscripten: "fn()> (32 bits)" -> "fn()> (64 bits)"

BTW there should be ignore-32bit / ignore-64bit / normalize-std???-32bit / normalize-std???-64bit so you don't need to write everything three times.

[nikomatsakis]

@kennytm yes, that sounds fine too. We'd have to make the "update references" script rather smarter (so it can parse those things out).

[nikomatsakis]

Maybe rewrite it in Rust :)

[kennytm]

@nikomatsakis since the files generated by compile-test is already normalized, the "update references" script shouldn't need to be changed.

[nikomatsakis]

@kennytm oh right, good point :)