meta(fuzzer): tracking issue for fuzzer improvements

[gakonst]

https://twitter.com/gakonst/status/1790770389523583163

Fuzz tests

Fuzzing-testing specific improvements, continuation of #4433 and #10190

UX/Features

High

• feat: Add "after-all" hook for testing #4300
• feat: support time-based and continuous fuzzing and invariant testing  #990
• feat(cheatcodes): support native bound cheatcode #8788
• feat(fuzz): add coverage guided fuzzing for stateless tests (currently only invariant mode works) #10877: stateless fuzzing support and additional ABI mutations

Nice to have

• feat(cheatcodes): add ability to exclude certain custom errors and revert reason strings from failing tests #4271
• feat(fuzz): generate solidity regression tests from failures #8117
• Ability to get sorted arrays when fuzzing #4097
• feat: fuzz corpus saving and replay in standard format #2552 - change existing format to standard when available, see Allow Echidna & Medusa to share the same corpus crytic/medusa#234
• Console logs should ideally print _during_ fuzzing, not just after testing is complete #3844

Bugs

• bug(forge test): --fail-fast flag does not work as it is not applied across multiple test suites #6529
• Invalid Enum value when fuzzing #6623
• bug: state appears to be shared between tests when linked libraries are used #8639

Invariants

Invariant-testing specific improvements, continuation of #4438

UX/Features

High

• feat(forge): Add internal metrics capability #3607
• Allow forge's contract invariant testing to contribute to coverage #4007
• forge test doesn't utilize all available threads for fuzzing/invariants #8898: share corpus and run as many invariants in as many threads as possible
• feat(forge test): add an option to continue fuzzing run on assertion failure #9727: ignore crashes to allow continuous fuzzing
• feat(invariant): support fuzz with random msg.value #8644: fuzz msg.value
• feat(fuzz): create initial seed corpus from forge test traces #10875: seed corpus from tests
• Using AST to seed the fuzzer dictionary #10233: insert constants and evaluated constant expressions in source in to fuzzer dictionary
• feat(forge): coverage guided fuzzing & time based campaigns for invariant mode #10190 (comment) (Maybe no longer needed: optimize the data structure of the corpus for lookups)
• implement compile-time, non-colliding instrumentation like afl++ PCGUARD in Solar and coverage-guided fuzzing to use it
• feat(invariant): add Optimization mode to Invariant Testing similar to Echidna #12190

Nice to have

• More granular control on invariant simulations #5018
• feat(invariant): extend the export of failed case to include traces as well #8114
• feat: vm.depth() cheatcode to return the depth of the current invariant run #2985
• Add invariant testing filter for excludeSelectors() #4352
• max_test_reject_rate: set a maximum test rejection rate per test function #4091
• feat: more flexible/powerful ways to define and test invariants #3452
• feat: test for reentrancy in invariant tests #1578
• feat(fuzz): structured logging for monitoring long-running fuzzing campaigns #10876: campaign stats logging
  • add gas/s

Performance

High

• feat(invariant): use storage layout to fuzz values from state by type  #8116
• feat(fuzz): do not populate dictionary with bytecode metadata #8115
• feat(forge): exclude precompiles by default in invariant tests #4287
• Built-in contracts like vm and the create2 factory should be excluded senders in invariants #4163
• feat: weight invariant selectors by number of selectors, not number of contracts. #2986

Benchmarks

High

• perf: fuzz/invariant benchmarks #3411
  • set up daily runner of https://github.com/grandizzy/fuzz-benchmarks/ + add more tests
• see details in feat: invariant benchmarks #7610
  • run Feedback on fuzzer benchmarking setup #4590
  • report as suggested in https://github.com/fuzz-evaluator/guidelines

Symbolic execution

• Symbolic Rust EVM #15

Mutation testing

• tracking: add mutation testing support #478

[zerosnacks]

this would be neat: #4000

[grandizzy]

this would be neat: #4000

yep, indeed! made a PR for it here #8338

[0xalpharush]

Following up on #10190

• share corpus and run as many invariants in as many threads as possible forge test doesn't utilize all available threads for fuzzing/invariants #8898
• ignore crashes to allow continuous fuzzing feat(forge test): add an option to continue fuzzing run on assertion failure #9727
• campaign stats logging feat(fuzz): structured logging for monitoring long-running fuzzing campaigns #10876
  • add gas/s
• stateless fuzzing support and additional ABI mutations feat(fuzz): add coverage guided fuzzing for stateless tests (currently only invariant mode works) #10877
• fuzz msg.value feat(invariant): support fuzz with random msg.value #8644
• seed corpus from tests feat(fuzz): create initial seed corpus from forge test traces #10875
• insert constants and evaluated constant expressions in source in to fuzzer dictionary Using AST to seed the fuzzer dictionary #10233
• Updated "favored" entries in corpus to retain all unique edges feat(fuzz): WorkerCorpus for multiple worker threads #11769 (comment)
• Maybe no longer needed: optimize the data structure of the corpus for lookups feat(forge): coverage guided fuzzing & time based campaigns for invariant mode #10190 (comment)
• Add ABI value shrinking to shrink implementation
• implement compile-time, non-colliding instrumentation like afl++ PCGUARD in Solar and coverage-guided fuzzing to use it