meta(invariants): tracking issue for improvements to invariant testing

[mds1]

Component

Forge

Describe the feature you would like

This issue tracks invariant-testing specific improvements. General fuzzer improvements are in #4433.

Items are roughly ordered by my opinion of priority, both at the header-level and the bullet-level. I say "roughly ordered" because some items are much bigger scope than others so having a strict ordering doesn't make much sense.

High Priority

These would all give a big improvement to invariant testing UX and performance, and should not be too hard to implement

UX/Features

• feat(forge test): add an option to continue fuzzing run on assertion failure #9727
• feat: commit state changes to test contract during invariant/fuzz tests #3005
• Function-level assertions don't work when using invariants. #4718
• feat: vm.depth() cheatcode to return the depth of the current invariant run #2985
• feat: weight invariant selectors by number of selectors, not number of contracts. #2986
• Add invariant testing filter for excludeSelectors() #4352

Bugs

• bug: invariant test log/traces not shown #2962
• bug(forge): custom errors are not decoded in invariant test traces #4178
• feat(forge): exclude precompiles by default in invariant tests #4287
• Built-in contracts like vm and the create2 factory should be excluded senders in invariants #4163
• test_invariant_shrink fails for ubuntu but works on mac #4381
• bug(forge): call_override to have same size as the number of (unsafe) external calls of the sequence #4245

Misc

• Confusing users by creating a dichotomy between fuzz testing and invariant testing #4162 — The terms we use are misleading so this worth clarifying / maybe renaming things a bit

Medium Priority

• feat(forge): enable console logging in invariant handlers #4951
• feat: add real-time counter for fuzz/invariant runs #585
• feat: vm.assume support in invariant tests #4190
• feat: test for reentrancy in invariant tests #1578 — More thought is needed around how to detect reentrancy issues, so this one is likely more work than all of the above, but it could be very useful. Related is bug(forge): call_override to have same size as the number of (unsafe) external calls of the sequence #4245

Low Priority

• Allow parametrised invariants #4834
• feat: Add "after-all" hook for testing #4300 — Listed as low because you can replicate this by calling functions at the end of each test
• feat: more flexible/powerful ways to define and test invariants #3452 — Needs more thought, but could be powerful
• feat(forge): Add internal metrics capability #3607 — An invariantCallSummary test and similar approaches are decent workarounds, though this still would be valuable

Additional context

No response

[zerosnacks]

Noticing a bit of flakyness with invariant tests recently in the CI, for example see: https://github.com/foundry-rs/foundry/actions/runs/10298801797/job/28504775878 / https://github.com/foundry-rs/foundry/actions/runs/10286582825/job/28467603861

cc @grandizzy

[grandizzy]

Noticing a bit of flakyness with invariant tests recently in the CI, for example see: https://github.com/foundry-rs/foundry/actions/runs/10298801797/job/28504775878 / https://github.com/foundry-rs/foundry/actions/runs/10286582825/job/28467603861

cc @grandizzy

yep, this should make it faster and avoid flakyness 🤞 #8634