meta(fuzzer): tracking issue for improvements to property tests and invariant tests #4433
Description
Component
Forge
Describe the feature you would like
This issue tracks general fuzzer improvements that improve both property tests and invariant tests. (There will be a separate tracking issue for things that are strictly invariant test related).
Items are roughly ordered by my opinion of priority, both at the header-level and the bullet-level. I say "roughly ordered" because some items are much bigger scope than others so having a strict ordering doesn't make much sense.
Fuzzer Benchmark/Tests
First priority is some tests and benchmarks. to verify behavior of the fuzzer has not regressed as it's changed. Potential test cases:
- condition not caught by invariant/fuzzer #2851
- https://github.com/crytic/echidna/tree/master/tests/solidity
- https://blog.trailofbits.com/2023/02/27/reusable-properties-ethereum-contracts-echidna/
This is tracked in a separate issue here: #3411
Refactor / Tech Debt / Cleanup
- Remove proptest? It has some limitations and isn't currently maintained (ref discussions in feat: support time-based and continuous fuzzing and invariant testing #990 and Is Proptest abandoned? proptest-rs/proptest#268). This would be a big change so shouldn't be done until we have tests/benchmarks in place.
- Deprecation of
fuzz.max_global_rejectsconfig #3153
Features / Performance
High Priority
- feat (proposal): UX for granularity over fuzz, invariant, and solidity sections #3062 OR Selective fuzz runs with NatSpec comments "@custom:fuzz-runs NUMBER" #4085 — Before implementing, need to decide if we like the config file approach or a comment-based approach, then finalize UX, then implement. Comment approach does add flexibility and allows future expansion to things like Ability to get sorted arrays when fuzzing #4097
- feat: deduplicate fuzz inputs #3521
- feat: add real-time counter for fuzz/invariant runs #585
Low priority:
- feat: add config option to turn off shrinking #4867
- feat: support time-based and continuous fuzzing and invariant testing #990 — Not supported by proptest so currently hard, but may be easy after a refactor. Low priority because it can be replicated by setting runs to max uint
vm.canRevert(): allow tests to revert in expected cases #4090- Ability to get sorted arrays when fuzzing #4097 — If we go with the comment approach for #3062, this is doable
- Console logs should ideally print _during_ fuzzing, not just after testing is complete #3844 — I think this would be hard to implement with the current architecture since things run in parallel, and you can accomplish the same thing with
vm.writeLine, so this isn't necessary IMO. May be worth closing max_test_reject_rate: set a maximum test rejection rate per test function #4091 — This flag would conflict with the existingmax_test_rejectsflag, so one needs to take precedence and we need warn if the both exist. Personally this feels like the complexity/config growth isn't worth it since the same functionality can already be accomplished, so may be worth closing
Failure Persistence and Replay
I have this last because the functionality can be replicated a bit tediously by copying failed tests into concrete tests.
Long Term Fuzz Techniques
Some more advanced techniques to consider down the road:
- Using outputs from symbolic execution to seed the fuzzer
- Techniques described in the discussion in Progress on Invariant Tests #3277
Additional context
No response