Skip to content

Conversation

@Gabriel-Trintinalia
Copy link
Contributor

@Gabriel-Trintinalia Gabriel-Trintinalia commented Apr 28, 2025

Description

This pull request introduces a new feature to enable saving only pre-merge block headers during synchronisation, along with related enhancements to the synchronisation pipeline and blockchain storage.

New Feature: Save Pre-Merge Headers Only for Snap Sync

  • Added a new CLI option --Xsnapsync-synchronizer-pre-merge-headers-only-enabled to enable saving only pre-merge block headers during synchronization
  • Introduced the SavePreMergeHeadersStep class to handle the logic for storing pre-merge headers in the blockchain.
    • Pre-merge block headers are stored and excluded from subsequent processing stages.
    • Post-merge block headers are passed through to the next processing stage.

Blockchain Storage

  • Added a new method unsafeStoreHeader in MutableBlockchain to store block headers updating the chain state.

Synchronisation Pipeline Updates

  • Updated FastSyncDownloadPipelineFactory to include SavePreMergeHeadersStep in the pipeline when the new feature is enabled.

Configuration Update

  • Updated Sepolia checkpoint to the merge block

Pipeline before the changes:

image
(headers: blue | bodies: red)

Pipeline with this PR:

image
(headers: blue | bodies: red)

  • Headers will be downloaded and saved until checkpoint block.
  • After checkpoint, bodies and receipts will be saved as usual

Disk usage

Node Name Checkpoint Block Number Blob Blockchain Space
8582-headers-ctl-1 n/a 404 GiB
8582-headers-ctl-2 n/a 404 GiB
8582-headers-ctl-3 n/a 404 GiB
8582-headers-ctl-4 n/a 404 GiB
8582-headers-only-1 1,735,371 403 GiB
8582-headers-only-2 1,735,371 403 GiB
8582-headers-only-3 1,735,371 403 GiB
8582-headers-only-4 1,735,371 403 GiB
8582-sync-near-head-1 8,000,000 15 GiB
8582-sync-near-head-2 8,000,000 15 GiB

Syncing logs:

2025-04-27 09:13:05.264+10:00 | EthScheduler-Services-40 (saveHistoricalHeaders) | INFO  | SavePreMergeHeadersStep | Pre-merge block headers import progress: 100 of 50000 (0.20%)
2025-04-27 09:13:35.420+10:00 | EthScheduler-Services-40 (saveHistoricalHeaders) | INFO  | SavePreMergeHeadersStep | Pre-merge block headers import progress: 21500 of 50000 (43.00%)
2025-04-27 09:14:03.865+10:00 | EthScheduler-Services-40 (saveHistoricalHeaders) | INFO  | SavePreMergeHeadersStep | Pre-merge block headers import completed at block 50000 (0x0e30a7c0c1cee426011e274abc746c1ad3c48757433eb0139755658482498aa9)
2025-04-27 09:14:05.711+10:00 | EthScheduler-Services-43 (importBlock) | INFO  | ImportBlocksStep | Block import progress: 50111 of 22352512 (0%), Peer count: 2
2025-04-27 09:14:35.951+10:00 | EthScheduler-Services-43 (importBlock) | INFO  | ImportBlocksStep | Block import progress: 76800 of 22352512 (0%), Peer count: 2

Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
@Gabriel-Trintinalia Gabriel-Trintinalia added syncing history reduce disk reqs thru history mgmt labels Apr 28, 2025
@Gabriel-Trintinalia Gabriel-Trintinalia changed the title feat(history expire): naive approach to save only headers for pre-merge blocks feat(history-expiry): naive approach to save only headers for pre-merge blocks Apr 28, 2025
Copy link
Contributor

@pinges pinges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor comment :-)

@Gabriel-Trintinalia Gabriel-Trintinalia enabled auto-merge (squash) April 29, 2025 05:43
Copy link
Contributor

@siladu siladu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the testing and write up 👍


private long getPreMergeHeaderBlockNumber(final SyncState syncState) {
return syncConfig.isSnapSyncSavePreMergeHeadersOnlyEnabled()
? syncState.getCheckpoint().map(checkpoint -> checkpoint.blockNumber() - 1).orElse(0L)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially a breaking change if any private networks make use of the existing checkpoint config?

We're kind of overloading the checkpoint concept, which I think makes sense for public networks as long as we're explicit about it. We should check usage for private.
In full 4444s, with rolling window, I guess this static checkpoint goes away entirely for public nets?

@siladu
Copy link
Contributor

siladu commented Apr 29, 2025

Missing changelog too - unless that's intentional?

Signed-off-by: Gabriel-Trintinalia <[email protected]>
Signed-off-by: Gabriel-Trintinalia <[email protected]>
@Gabriel-Trintinalia Gabriel-Trintinalia changed the title feat(history-expiry): naive approach to save only headers for pre-merge blocks feat(history-expiry): save only headers for pre-merge blocks Apr 29, 2025
Copy link
Contributor

@siladu siladu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just the changelog missing potentially, especially if we want to mention breaking changes (e.g. to private networks) at this point.

@Gabriel-Trintinalia Gabriel-Trintinalia enabled auto-merge (squash) April 30, 2025 04:16
@Gabriel-Trintinalia Gabriel-Trintinalia merged commit 1db0313 into hyperledger:main Apr 30, 2025
41 checks passed
@siladu siladu moved this to Done in History Expiry May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

history reduce disk reqs thru history mgmt mainnet snapsync syncing

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants