[Meta] CI speed

[dannywillems]

This meta-issue will gather all the findings regarding the speed of the CI. It is not titled "CI latency" or "CI improvement" as it is meant to stay open and keep track of all discussions.

Monitoring: https://github.com/o1-labs/openmina/actions/metrics/performance?tab=jobs&filters=workflow_file_name%3Aci.yaml

[dannywillems]

This workflow can be considered as an example: https://github.com/o1-labs/openmina/actions/runs/16959993301.

We do have a pretty fast CI with caching already, but i'm worried about these two. I would like to give a shot to nextest after 1307 (targets in the makefile already added, but I will test locally first), or in a follow-up PR.

scenario e2e tests are taking a relatively long time too, but nothing at the CI level we can do.

and same for p2p and record-replay. Nothing at the CI level.

Building should get faster. Let's see if it is faster with the incremental option (see #1307).

[dannywillems]

Idea from @0xMimir: adding timeouts

Also we should set timeouts in CI, like the job yesterday that was running for half a day, on this PR I think that last 4 pending checks will run until killed

[dannywillems]

Idea from @0xMimir: adding timeouts

Also we should set timeouts in CI, like the job yesterday that was running for half a day, on this PR I think that last 4 pending checks will run until killed

See #1310

[richardpringle]

TL;DR in the next comment

So, I have some thoughts here, and we are already more or less on the same page. There's one fact that we cannot change: compiling Rust takes time.

I believe we are already using the rust-cache action, so there's not much more we can do there.

Beyond that, tests should be broken down into three categories: unit tests, integration tests, and black-box tests. Fuzzing and property-based testing can happen in all three categories.

Unit tests should be quick. This number is arbitrary, but tests automatically have a warning when they take over 60s. So I would say we should aim to have no unit tests take over 60s. If they take longer than that, they should be run as a part of a separate CI job with a feature flag. Same goes for proptest. But these are only for tests that need private access to internals.

Integration tests can be longer running, but I would suggest that no one test take longer than 5 minutes. It's an arbitrary number, but I base this on the time you're willing to wait for CI. If not test takes longer than 5-minutes, then I would expect that CI not take longer than 15 minutes (5-8mins for compiling, 5-8mins for testing).

Black-box testing can take significantly longer. A black-box test could be the production of a new block.

Unit and integration tests should be run on every PR, but black-box tests should be run after merging. Why would we merge something if we don't know if it works end-to-end? Because open PRs are a maintenance burden. Often, if all the unit and integration tests are passing, but a black-box test is failing, this is more akin to a bug than a poorly implemented feature. It's often easier to add the fix afterwards. All black-box tests should be passing before we release, but shouldn't necessarily block PRs. Sometimes they can be disruptive, in which case we can revert the changes, but this isn't the most common case, usally the fix is going to be something relatively quick and small.

Thoughts?

The biggest advantage is actually for all the cases where small changes are unlikely to affect any major features that are black-box tested. These are the vast majority of PR. People don't want to wait around for 30-mins for a PR pass when they make a small change. They also don't want to start something new in those 30-minutes if they know they're going to have to go back.

This does mean however, that you want CI to complete on every new tip of a shared branch (develop in our case), so you don't want to cancel in-progress jobs.

[richardpringle]

AI generated TL;DR for above

Speed up CI, accepting that Rust compilation is slow. The main idea is to change when different types of tests are run:

1. Run on Every PR (Keep CI < 15 mins):

• Unit Tests: Aim for < 60s each.
• Integration Tests: Aim for < 5 mins each.

2. Run After Merging (on the develop branch):

• Black-box Tests: Run the slow, long-running end-to-end tests here.

Reasoning:

• This significantly speeds up the workflow for most PRs, which are small changes.
• Developers won't have to wait 30+ minutes for a simple change.
• Failing black-box tests after a merge can be treated like a bug and fixed, rather than blocking the PR.
• All tests must still pass before a release, but they don't need to block every single PR.