feat (proposal): UX for granularity over fuzz, invariant, and solidity sections

[mds1]

Component

Forge

Describe the feature you would like

Supersedes #744

For the [fuzz] and [invariant] sections, you must define configuration per-profile and it applies to all tests. Some tests need more/less fuzzing than others, so it would be great to configure this on a per-test basis. Below is a proposal for how to structure the foundry.toml to allow this.

Similarly, it would be great to allow different solc settings per contract (e.g. 1 optimizer run for MyBigContract but 10M optimizer runs for all others). This proposal would enable that as well.

# OVERVIEW
# Each fuzz/invariant section can have subprofiles. In these subprofiles you
# can define all the standard fuzz/invariant options, but additionally the same
# matching flags that `forge test` has are supported. This means good naming
# conventions are important to make targeting your tests easy.

# EXAMPLE
# Here is an example using fuzz configs to do the following:
#
# Default profile:
#   - 250 fuzz runs by default
#   - `low` tests execute 50 fuzz runs
#   - `high` tests execute 500 fuzz runs
# CI profile:
#   - 1000 fuzz runs by default
#   - `low` tests execute 500 fuzz runs
#   - `high` tests execute 5000 fuzz runs with 100_000 max rejections
#
# In this example, we only use the match-test flag, but any supported match
# flags can be used.

# First we define our default profile
[profile.default.fuzz]
runs = 250 # 250 runs by default for the default profile

[profile.default.fuzz.low]
# Any test that matches the provided regex uses this subprofile
match-test = "testFuzzLow.*"
runs = 50

[profile.default.fuzz.high]
match-test = "testFuzzHigh.*"
runs = 500

# Now we define our CI profile
[profile.ci.fuzz]
runs = 1000

[profile.ci.fuzz.low]
match-test = "testFuzzLow.*"
runs = 500

[profile.ci.fuzz.high]
match-test = "testFuzzHigh.*"
runs = 5000
max-global-rejects = 100_000

# EXTENSIONS
# You can imagine this extending to e.g. solidity configurations too when a
# [solidity] table is added, such as shown below:

# We define a lite mode to keep the optimizer off to speed up dev/testing.
# Since there's no `lite` fuzz config, it uses the default profile's settings.
[profile.lite.solidity]
optimizer = false

# In our default solidity profile we apply 10M optimizer runs.
[profile.default.solidity]
optimizer = true
optimizer-runs = 10_000_000

# But we have one really big contract, so we use 1 optimizer run to minimize
# its size.
[profile.default.solidity.size]
optimizer = true
optimizer-runs = 1
match-contract = "MyBigContract"

Additional context

No response

[gakonst]

To select the profile ci one would do FOUNDRY_CONFIG=ci forge t, how would one choose the fuzzer profile?

[mds1]

FOUNDRY_CONFIG=ci forge t would use the settings under profile.ci.fuzz for fuzz tests. Within that profile are sub profiles: profile.ci.fuzz is the default config for that profile, and in the above example there's also profile.ci.fuzz.low and profile.ci.fuzz.high. So forge would:

• Look at all match-* keys under profile.ci.fuzz.low, and apply the CI low config to those tests.
• Look at all match-* keys under profile.ci.fuzz.high, and apply the CI high config to those tests. (If there's a conflict, we should just go with whatever was processed last and print a warning so the user can fix it)
• All other tests use the default under profile.ci.fuzz.
• If nothing is specified under profile.ci.fuzz, it falls back to profile.default.fuzz which is an alias for fuzz

[rkrasiuk]

i've got mixed feelings about the config complexity this introduces. @mds1 wdtyt about keeping the sections packed & simply adding the overrides to [fuzz] & [invariant] sections

[fuzz]
runs = 250
overrides = { "testFuzzLow.*" = { runs = 50 }, "testFuzzHigh.*" = { runs = 500 }}

[profile.ci.fuzz]
runs = 1000
overrides = { "testFuzzLow.*" = { runs = 500 }, "testFuzzHigh.*" = { runs = 5000, max-global-rejects = 100_000 } }

[mds1]

Yea I feel the same way about the complexity, but couldn't think of a good alternative 😕

In general I like your overrides suggestion, but:

1. I think we should support matching by either test name, contract, or file, whereas that looks like just test name. This can be resolved by something like overrides-match-path, overrides-no-match-test, etc.
2. Inline tables for overrides can get long, and I'd want to split them onto multiple lines for readability. However the toml spec says:

Inline tables are intended to appear on a single line ... No newlines are allowed between the curly braces unless they are valid within a value. Even so, it is strongly discouraged to break an inline table onto multiples lines. If you find yourself gripped with this desire, it means you should be using standard tables.

Which was part of my motivation for the original format

[rkrasiuk]

yeah, makes sense. not a fan of long lines either hence mixed feelings. one more idea is to have a cheatcode for a contract or test to set the config values, however that'd be both cumbersome to implement and for the end user to track across the codebase

[mds1]

Yea I do think cheatcodes could be nice UX, see approach 4 in #744. However even that can get cumbersome since there's multiple parameters to set—we'd probably want a struct that defines all fuzz values and use some magic values to mean "I'm not setting this one, fallback to the default". But like you said it's probably significantly more work to implement and harder for users to track.

I think the original proposal feels complex and hard to read because everything is visually "flat", whereas if this were formatted as JSON it would be more readable due to the indentation changes. Therefore I think using some comments to visually separate profiles can make it more readable (inline tables would also help, though technically they're supposed to be limited to one line I believe):

# =================================
# ======== Default Profile ========
# =================================

# -------- General --------

[profile.default]
verbosity = 3

# -------- Solidity --------

[profile.lite.solidity]
optimizer = false

[profile.default.solidity]
optimizer = true
optimizer-runs = 10_000_000

[profile.default.solidity.size]
optimizer = true
optimizer-runs = 1
match-contract = "MyBigContract"

# -------- Fuzz --------

[profile.default.fuzz]
runs = 250 # 250 runs by default for the default profile

[profile.default.fuzz.low]
match-test = "testFuzzLow.*"
runs = 50

[profile.default.fuzz.high]
match-test = "testFuzzHigh.*"
runs = 500

# ====================
# ======== CI ========
# ====================

# -------- Fuzz --------

[profile.ci.fuzz]
runs = 1000

[profile.ci.fuzz.low]
match-test = "testFuzzLow.*"
runs = 500

[profile.ci.fuzz.high]
match-test = "testFuzzHigh.*"
runs = 5000
max-global-rejects = 100_000