[WIP] all: implement chain freezer #884
Open
+13,581
−925
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gzliudan
force-pushed
the
chain_freezer
branch
18 times, most recently
from
8339678 to
c2625df
Compare
gzliudan
force-pushed
the
chain_freezer
branch
12 times, most recently
from
a79e88b to
8275a94
Compare
While investigating ethereum#22374, I noticed that the Sync operation of the freezer does not take the table lock. It also doesn't call sync for all files if there is an error with one of them. I doubt this will fix anything, but didn't want to drop the fix on the floor either.
* core/rawdb: fix freezer validation * core/rawdb: address comment
This PR implements resettable freezer by adding a ResettableFreezer wrapper. The resettable freezer wraps the original freezer in a way that makes it possible to ensure atomic resets. Implementation wise, it relies on the os.Rename and os.RemoveAll to atomically delete the original freezer data and re-create a new one from scratch.
This PR fixes an issue which might result in data lost in freezer. Whenever mutation happens in freezer, all data will be written into head data file and it will be rotated with a new one in case the size of file reaches the threshold. Theoretically, the rotated old data file should be fsync'd to prevent data loss. In freezer.Sync function, we only fsync: (1) index file (2) meta file and (3) head data file. So this PR forcibly fsync the head data file if mutation happens in the boundary of data file.
This PR does a few things. It fixes a shutdown-order flaw in the chainfreezer. Previously, the chain-freezer would shutdown the freezer backend first, and then signal for the loop to exit. This can lead to a scenario where the freezer tries to fsync closed files, which is an error-conditon that could lead to exit via log.Crit. It also makes the printout more detailed when truncating 'dangling' items, by showing the exact number instead of approximate MB. This PR also adds calls to fsync files before closing them, and also makes the `db inspect` command slightly more robust.
The meter for "for measuring the effective amount of data read" within the freezertable was never updated. This change remedies that. --------- Signed-off-by: jsvisa <[email protected]>
This change adds the ability to perform reads from freezer without size limitation. This can be useful in cases where callers are certain that out-of-memory will not happen (e.g. reading only a few elements). The previous API was designed to behave both optimally and secure while servicing a request from a peer, whereas this change should _not_ be used when an untrusted peer can influence the query size.
Avoid truncating files, if ancients are opened in readonly mode. With this change, we return error instead of trying (and failing) to repair
This allows using the freezer from multiple processes at once in read-only mode. Co-authored-by: Martin Holst Swende <[email protected]>
This PR adds more error message for debugging purpose.
…. (28379) This adds warning logs when the read does not match the expected count. We can also remove the size limit since the function documentation explicitly states that callers should limit the count.
* core/rawdb: fsync the index and data file after each freezer write * core/rawdb: fsync the data file in freezer after write
…(28525) This is the fix to issue ethereum#27483. A new hiddenBytes() is introduced to calculate the byte size of hidden items in the freezer table. When reporting the size of the freezer table, size of the hidden items will be subtracted from the total size. --------- Co-authored-by: Yifan <Yifan Wang> Co-authored-by: Gary Rong <[email protected]>
…gapped (26719) This change prints out more information about the problem, in the case where geth detects a gap between leveldb and ancients, so we can determine more exactly where the gap is (what the first missing is). Also prints out more metadata. --------- Co-authored-by: Martin Holst Swende <[email protected]>
This pull request removes the `fsync` of index files in freezer.ModifyAncients function for performance gain. Originally, fsync is added after each freezer write operation to ensure the written data is truly transferred into disk. Unfortunately, it turns out `fsync` can be relatively slow, especially on macOS (see ethereum#28754 for more information). In this pull request, fsync for index file is removed as it turns out index file can be recovered even after a unclean shutdown. But fsync for data file is still kept, as we have no meaningful way to validate the data correctness after unclean shutdown. --- **But why do we need the `fsync` in the first place?** As it's necessary for freezer to survive/recover after the machine crash (e.g. power failure). In linux, whenever the file write is performed, the file metadata update and data update are not necessarily performed at the same time. Typically, the metadata will be flushed/journalled ahead of the file data. Therefore, we make the pessimistic assumption that the file is first extended with invalid "garbage" data (normally zero bytes) and that afterwards the correct data replaces the garbage. We have observed that the index file of the freezer often contain garbage entry with zero value (filenumber = 0, offset = 0) after a machine power failure. It proves that the index file is extended without the data being flushed. And this corruption can destroy the whole freezer data eventually. Performing fsync after each write operation can reduce the time window for data to be transferred to the disk and ensure the correctness of the data in the disk to the greatest extent. --- **How can we maintain this guarantee without relying on fsync?** Because the items in the index file are strictly in order, we can leverage this characteristic to detect the corruption and truncate them when freezer is opened. Specifically these validation rules are performed for each index file: For two consecutive index items: - If their file numbers are the same, then the offset of the latter one MUST not be less than that of the former. - If the file number of the latter one is equal to that of the former plus one, then the offset of the latter one MUST not be 0. - If their file numbers are not equal, and the latter's file number is not equal to the former plus 1, the latter one is valid And also, for the first non-head item, it must refer to the earliest data file, or the next file if the earliest file is not sufficient to place the first item(very special case, only theoretical possible in tests) With these validation rules, we can detect the invalid item in index file with greatest possibility. --- But unfortunately, these scenarios are not covered and could still lead to a freezer corruption if it occurs: **All items in index file are in zero value** It's impossible to distinguish if they are truly zero (e.g. all the data entries maintained in freezer are zero size) or just the garbage left by OS. In this case, these index items will be kept by truncating the entire data file, namely the freezer is corrupted. However, we can consider that the probability of this situation occurring is quite low, and even if it occurs, the freezer can be considered to be close to an empty state. Rerun the state sync should be acceptable. **Index file is integral while relative data file is corrupted** It might be possible the data file is corrupted whose file size is extended correctly with garbage filled (e.g. zero bytes). In this case, it's impossible to detect the corruption by index validation. We can either choose to `fsync` the data file, or blindly believe that if index file is integral then the data file could be integral with very high chance. In this pull request, the first option is taken.
Fixes an issue where the node panics when an LStat fails with something other than os.ErrNotExist closes ethereum#30968
This is a follow-up PR to ethereum#29792 to get rid of the data file sync. **This is a non-backward compatible change, which increments the database version from 8 to 9**. We introduce a flushOffset for each freezer table, which tracks the position of the most recently fsync’d item in the index file. When this offset moves forward, it indicates that all index entries below it, along with their corresponding data items, have been properly persisted to disk. The offset can also be moved backward when truncating from either the head or tail of the file. Previously, the data file required an explicit fsync after every mutation, which was highly inefficient. With the introduction of the flush offset, the synchronization strategy becomes more flexible, allowing the freezer to sync every 30 seconds instead. The data items above the flush offset are regarded volatile and callers must ensure they are recoverable after the unclean shutdown, or explicitly sync the freezer before any proceeding operations. --------- Co-authored-by: Felix Lange <[email protected]>
This PR addresses a flaw in the freezer table upgrade path. In v1.15.0, freezer table v2 was introduced, including an additional field (`flushOffset`) maintained in the metadata file. To ensure backward compatibility, an upgrade path was implemented for legacy freezer tables by setting `flushOffset` to the size of the index file. However, if the freezer table is opened in read-only mode, this file write operation is rejected, causing Geth to shut down entirely. Given that invalid items in the freezer index file can be detected and truncated, all items in freezer v0 index files are guaranteed to be complete. Therefore, when operating in read-only mode, it is safe to use the freezer data without performing an upgrade.
Here we add the notion of prunable tables for the `TruncateTail` operation in the freezer. TruncateTail for the chain freezer now only truncates the body and receipts tables, leaving headers and hashes as-is. This change also requires changing the validation/repair at startup to allow for tables with different tail. For the header and hash tables, we now require them to start at number zero. --------- Co-authored-by: Felix Lange <[email protected]> Co-authored-by: Gary Rong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
This PR is part of the effort to solve issue #570, it:
BlockChainVersionfrom 6 to 8TODO:
Ref:
Types of changes
What types of changes does your code introduce to XDC network?
Put an
✅in the boxes that applyImpacted Components
Which part of the codebase this PR will touch base on,
Put an
✅in the boxes that applyChecklist
Put an
✅in the boxes once you have confirmed below actions (or provide reasons on not doing so) that