Problem: no command to fix corrupted data in versiondb #1685
Conversation
Closes: crypto-org-chain#1683 Solution: - add fix command to fix corrupted data in versiondb
WalkthroughThe changes in this pull request include updates to the Changes
Assessment against linked issues
Possibly related PRs
Suggested reviewers
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1685 +/- ##
==========================================
+ Coverage 35.26% 35.47% +0.21%
==========================================
Files 123 124 +1
Lines 11755 11893 +138
==========================================
+ Hits 4145 4219 +74
- Misses 7196 7251 +55
- Partials 414 423 +9
|
yihuang
requested review from
leejw51crypto and
thomas-nguy
and removed request for
a team
…pto-org-chain#1685) Closes: crypto-org-chain#1683 Solution: - add fix command to fix corrupted data in versiondb rename support SkipVersionZero support SkipVersionZero cleanup
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Outside diff range and nitpick comments (7)
app/versiondb.go (1)
27-44: Consider startup sequence and recovery scenarios.The current implementation introduces critical data fixing and version handling functionality. Please consider:
Startup Sequence:
- Ensure proper ordering of operations (fix data → skip version zero → normal operations)
- Document the startup sequence for operators
Recovery Scenarios:
- Document recovery procedures if the fix process fails
- Consider adding a dry-run mode for the fix operation
- Add logging for fixed entries
Monitoring:
- Consider adding metrics for:
- Number of entries fixed
- Number of version zero entries encountered
- Fix operation duration
Would you like assistance in implementing any of these suggestions?
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 28-31: app/versiondb.go#L28-L31
Added lines #L28 - L31 were not covered by tests
[warning] 39-40: app/versiondb.go#L39-L40
Added lines #L39 - L40 were not covered by tests
[warning] 43-44: app/versiondb.go#L43-L44
Added lines #L43 - L44 were not covered by testsversiondb/tsrocksdb/opts.go (2)
67-67: Enhance function documentation for clarity.The current documentation could be more detailed to explain:
- When to use read-only mode vs. regular mode
- The significance of the
errorIfWalFileExistsparameter- How this relates to the data corruption fix process
Consider expanding the documentation like this:
-// OpenVersionDBForReadOnly opens versiondb for read-only. +// OpenVersionDBForReadOnly opens versiondb in read-only mode, which is useful for +// safely inspecting database contents, especially when dealing with potentially +// corrupted data. The errorIfWalFileExists parameter determines whether to return +// an error if WAL (Write-Ahead Log) files exist, which could indicate incomplete +// transactions.
67-78: Consider documenting the recovery procedure.The read-only functionality is a crucial part of the data corruption fix. To ensure proper usage:
- Document the complete recovery procedure in the package documentation
- Add integration tests that demonstrate the fix workflow:
- Opening in read-only mode
- Identifying corrupted data
- Applying fixes
Would you like me to help create:
- A detailed recovery procedure document?
- Integration test examples?
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 68-77: versiondb/tsrocksdb/opts.go#L68-L77
Added lines #L68 - L77 were not covered by testsversiondb/tsrocksdb/iterator.go (1)
129-140: Consider adding loop protection in trySkipNonZeroVersion.While the implementation is correct, there's a potential risk of infinite loops if all remaining items have timestamp zero. Consider adding a safety check.
func (itr rocksDBIterator) trySkipNonZeroVersion() { if itr.skipVersionZero { + maxIterations := 1000 // Adjust based on expected data size + iterations := 0 for itr.Valid() && itr.timestamp() == 0 { + iterations++ + if iterations > maxIterations { + panic("Possible infinite loop detected while skipping version zero entries") + } itr.Next() } } }versiondb/tsrocksdb/store_test.go (1)
159-228: Test implementation looks good, consider adding edge cases.The test effectively validates the core functionality of skipping version zero and fixing corrupted data. However, consider enhancing it with additional test cases:
- Edge cases:
- Empty values
- Multiple corrupted entries
- Boundary conditions for version numbers
- Verification of internal state after fix
Here's a suggested addition to strengthen the test coverage:
err = store.FixData([]types.StoreKey{storeKey}) require.NoError(t, err) bz, err = store.GetAtVersion(storeKey.Name(), key2, &i) require.NoError(t, err) require.Equal(t, []byte{2}, bz) + + // Test edge cases + emptyKey := []byte("empty") + emptyKeyWrong := cloneAppend(emptyKey, wrongTz[:]) + + // Test empty value + err = store.PutAtVersion(0, []*types.StoreKVPair{ + {StoreKey: storeKey.Name(), Key: emptyKeyWrong, Value: []byte{}}, + }) + require.NoError(t, err) + + // Verify fix handles empty values + err = store.FixData([]types.StoreKey{storeKey}) + require.NoError(t, err) + + bz, err = store.GetAtVersion(storeKey.Name(), emptyKey, &i) + require.NoError(t, err) + require.Empty(t, bz)versiondb/tsrocksdb/store.go (2)
44-44: [Nitpick] Avoid including external issue links in code commentsIncluding external issue links like
https://github.com/crypto-org-chain/cronos/issues/1683in code comments may not be ideal, especially if the repository becomes private or the issue is moved. Consider summarizing the issue or referencing it in documentation instead.
281-281: [Refactor Suggestion] Verify loop continuation logic is testedThe
continuestatement at line 281 is not covered by tests. Ensure that there are unit tests wherecleanevaluates totrueto validate that the loop behaves correctly in this case.🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 281-281: versiondb/tsrocksdb/store.go#L281
Added line #L281 was not covered by tests
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (6)
CHANGELOG.md(1 hunks)app/versiondb.go(1 hunks)versiondb/tsrocksdb/iterator.go(5 hunks)versiondb/tsrocksdb/opts.go(1 hunks)versiondb/tsrocksdb/store.go(6 hunks)versiondb/tsrocksdb/store_test.go(2 hunks)
🧰 Additional context used
🪛 GitHub Check: codecov/patch
app/versiondb.go
[warning] 28-31: app/versiondb.go#L28-L31
Added lines #L28 - L31 were not covered by tests
[warning] 39-40: app/versiondb.go#L39-L40
Added lines #L39 - L40 were not covered by tests
[warning] 43-44: app/versiondb.go#L43-L44
Added lines #L43 - L44 were not covered by tests
versiondb/tsrocksdb/opts.go
[warning] 68-77: versiondb/tsrocksdb/opts.go#L68-L77
Added lines #L68 - L77 were not covered by tests
versiondb/tsrocksdb/store.go
[warning] 103-104: versiondb/tsrocksdb/store.go#L103-L104
Added lines #L103 - L104 were not covered by tests
[warning] 241-242: versiondb/tsrocksdb/store.go#L241-L242
Added lines #L241 - L242 were not covered by tests
[warning] 253-254: versiondb/tsrocksdb/store.go#L253-L254
Added lines #L253 - L254 were not covered by tests
[warning] 264-265: versiondb/tsrocksdb/store.go#L264-L265
Added lines #L264 - L265 were not covered by tests
[warning] 274-275: versiondb/tsrocksdb/store.go#L274-L275
Added lines #L274 - L275 were not covered by tests
[warning] 281-281: versiondb/tsrocksdb/store.go#L281
Added line #L281 was not covered by tests
🔇 Additional comments (10)
app/versiondb.go (3)
38-40:
Critical functionality requires test coverage.
This code segment implements the fix for corrupted data (issue #1683), which is a critical functionality. However, it lacks test coverage. Given its importance in fixing database corruption, comprehensive testing is essential.
#!/bin/bash
# Check for existing tests of FixData functionality
echo "Checking for FixData tests..."
rg -A 5 "TestFixData|test.*FixData" "versiondb/"
# Check for any existing error handling tests
echo "Checking for error handling tests..."
rg -A 5 "TestSetupVersionDBError|test.*VersionDBError" "app/"Would you like assistance in:
- Generating comprehensive test cases for the data fix functionality?
- Creating error scenario tests?
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 39-40: app/versiondb.go#L39-L40
Added lines #L39 - L40 were not covered by tests
43-44: 🛠️ Refactor suggestion
Verify persistence of skipVersionZero setting.
While enabling version zero skipping is correct, please ensure:
- This setting persists across database restarts
- The setting is applied before any data operations occur
- Test coverage is added for this configuration
#!/bin/bash
# Check for persistence handling in the codebase
echo "Checking for persistence handling..."
rg -A 5 "SetSkipVersionZero|skipVersionZero" "versiondb/"
# Check for existing tests
echo "Checking for skip version tests..."
rg -A 5 "TestSkipVersionZero|test.*SkipVersion" "versiondb/"Consider adding a configuration option to make this setting configurable and persistent.
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 43-44: app/versiondb.go#L43-L44
Added lines #L43 - L44 were not covered by tests
27-31: Add test coverage for exposed keys preparation.
While the logic is correct, this new code segment lacks test coverage. Consider adding test cases to verify the proper collection of store keys.
Would you like assistance in generating test cases for this functionality?
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 28-31: app/versiondb.go#L28-L31
Added lines #L28 - L31 were not covered by tests
versiondb/tsrocksdb/iterator.go (4)
16-18: LGTM! Well-documented field addition.
The new skipVersionZero field is well-placed and documented with a link to the relevant issue, providing clear context for future maintainers.
125-127: LGTM! Correct placement of version check.
The call to trySkipNonZeroVersion is properly placed after the standard iteration, ensuring consistent version filtering behavior.
164-167: Verify nil handling in all slice processing methods.
The nil check addition is a good defensive programming practice. Let's verify similar checks exist in other slice handling methods.
#!/bin/bash
# Description: Look for other slice handling methods that might need nil checks
# Search for methods handling grocksdb.Slice
ast-grep --pattern 'func $_($_: *grocksdb.Slice) $_'23-23: Verify constructor usage across the codebase.
The constructor changes look good and properly initialize the new field. The immediate call to trySkipNonZeroVersion ensures consistent state from creation.
Also applies to: 46-57
✅ Verification successful
Constructor usage is properly updated across the codebase
The verification shows that:
- The only call to
newRocksDBIteratoris instore.goand correctly passes theskipVersionZeroparameter from the store's field - The
skipVersionZerofield is properly defined in both theStoreandrocksDBIteratorstructs - No other usages of the constructor exist in the codebase
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify all calls to newRocksDBIterator have been updated with the skipVersionZero parameter
# Search for constructor calls
rg "newRocksDBIterator\(" --type go
Length of output: 54
Script:
#!/bin/bash
# Let's try a more comprehensive search for the constructor usage
rg "newRocksDBIterator" -A 2 -B 2 --type go
# Also search for any potential interface or struct definitions related to this
ast-grep --pattern 'type $_ struct {
$$$
skipVersionZero bool
$$$
}'
# Search for any test files that might use this constructor
fd -e go -E vendor --exec grep -l "newRocksDBIterator" {}
Length of output: 1870
versiondb/tsrocksdb/store_test.go (2)
7-8: LGTM!
The new imports are necessary and correctly placed.
230-242: LGTM!
The helper types and functions are well-implemented:
kvPairprovides a clean structure for test resultsconsumeIteratorproperly manages iterator lifecycle
CHANGELOG.md (1)
8-9: LGTM! The changelog entries are well-formatted and accurate.
The new entries properly document:
- The bug fix for corrupted versiondb data (#1685)
- The improvement to versiondb KVStore interface (#1684)
The entries follow the changelog conventions and clearly describe the changes.
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
versiondb/tsrocksdb/store.go(6 hunks)
🧰 Additional context used
🪛 GitHub Check: codecov/patch
versiondb/tsrocksdb/store.go
[warning] 103-104: versiondb/tsrocksdb/store.go#L103-L104
Added lines #L103 - L104 were not covered by tests
[warning] 241-242: versiondb/tsrocksdb/store.go#L241-L242
Added lines #L241 - L242 were not covered by tests
[warning] 253-254: versiondb/tsrocksdb/store.go#L253-L254
Added lines #L253 - L254 were not covered by tests
[warning] 264-265: versiondb/tsrocksdb/store.go#L264-L265
Added lines #L264 - L265 were not covered by tests
[warning] 275-276: versiondb/tsrocksdb/store.go#L275-L276
Added lines #L275 - L276 were not covered by tests
[warning] 283-283: versiondb/tsrocksdb/store.go#L283
Added line #L283 was not covered by tests
🔇 Additional comments (3)
versiondb/tsrocksdb/store.go (3)
42-44: LGTM: Store struct changes are well-documented
The addition of skipVersionZero field and its setter method are straightforward and properly documented with a reference to the issue.
Also applies to: 65-67
159-169: LGTM: Good consolidation of iterator logic
The consolidation of forward and reverse iteration logic into a single method improves maintainability.
248-290: Verify data consistency after fix
The implementation looks correct and properly handles resources. However, given the critical nature of data fixing:
- Consider adding logging to track the progress and number of fixed entries
- Verify that the fix doesn't affect any valid version 0 entries
✅ Verification successful
Logging recommended, but version 0 handling is safe
The code analysis shows that:
- The
fixDataStorefunction is the only implementation handling version 0 data migration - The timestamp handling in tests confirms proper versioning behavior
- The fix is safe as it:
- Only modifies entries where the value differs from the original
- Preserves the original timestamps
- Uses proper resource cleanup
Adding logging would be beneficial for operational visibility, but it's not a correctness concern. The implementation ensures data consistency by comparing values before making any changes.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for any remaining version 0 entries after fix
# This helps verify the completeness of the fix
# Search for any remaining version 0 entries in the codebase
rg -l "version.*0.*ts\[:" .
# Search for any explicit version 0 handling in tests
rg -l "version.*=.*0" . | grep "_test\.go"
Length of output: 74
Script:
#!/bin/bash
# Let's analyze the codebase more thoroughly to understand version 0 handling
# Search for version 0 related code and tests
rg -A 5 "version.*0" versiondb/
# Look for timestamp handling in tests
rg -A 5 "TimestampSize|timestamp" versiondb/ | grep -A 5 "_test.go"
# Check for other data fixing or migration functions
ast-grep --pattern 'func $_($_ Store) fix$_($$$) error {
$$$
}'
Length of output: 4769
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 253-254: versiondb/tsrocksdb/store.go#L253-L254
Added lines #L253 - L254 were not covered by tests
[warning] 264-265: versiondb/tsrocksdb/store.go#L264-L265
Added lines #L264 - L265 were not covered by tests
[warning] 275-276: versiondb/tsrocksdb/store.go#L275-L276
Added lines #L275 - L276 were not covered by tests
[warning] 283-283: versiondb/tsrocksdb/store.go#L283
Added line #L283 was not covered by tests
Signed-off-by: yihuang <[email protected]>
Co-authored-by: mmsqe <[email protected]> Signed-off-by: yihuang <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Outside diff range and nitpick comments (3)
versiondb/client/fixdata.go (1)
47-50: Add progress reporting for long operationsThe
FixDataoperation could be time-consuming for large databases. Consider adding progress reporting.Consider wrapping the operation with progress reporting:
versionDB := tsrocksdb.NewStoreWithDB(db, cfHandle) + cmd.Printf("Starting data fix for stores: %v\n", stores) if err := versionDB.FixData(stores, dryRun); err != nil { return err } + cmd.Printf("Data fix completed successfully\n")versiondb/tsrocksdb/store.go (2)
309-310: Address the FIXME comment.The FIXME comment references issue #1689. Please ensure this issue is tracked and resolved.
Would you like me to help create a GitHub issue to track this TODO?
314-316: Improve error message for invalid key length.The error message could be more descriptive by including the expected minimum length.
Apply this diff to improve the error message:
- return nil, fmt.Errorf("invalid key length: %X, store: %s", key, storeName) + return nil, fmt.Errorf("invalid key length %d (expected >= %d) for key %X in store %s", len(key), TimestampSize, key, storeName)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
versiondb/client/fixdata.go(1 hunks)versiondb/tsrocksdb/store.go(7 hunks)
🔇 Additional comments (6)
versiondb/client/fixdata.go (2)
14-59: Add test coverage for the command
The command lacks test coverage as noted in previous reviews. Tests should verify:
- Command execution with valid/invalid arguments
- Error handling scenarios
- Integration with versionDB.FixData
Would you like me to help create comprehensive test cases for this command?
38-42: Verify database access modes
Let's verify that the database access modes are correctly implemented in the referenced functions.
✅ Verification successful
Database access modes are correctly implemented
The verification confirms that both functions are properly implemented:
OpenVersionDBusesOpenDbColumnFamiliesfor read-write access with proper column family setupOpenVersionDBForReadOnlyusesOpenDbForReadOnlyColumnFamiliesfor read-only access with the same column family configuration
The usage in the code snippet is correct, selecting the appropriate access mode based on the dryRun flag.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check implementation of OpenVersionDBForReadOnly and OpenVersionDB
# Search for the implementation of these functions
ast-grep --pattern 'func OpenVersionDBForReadOnly($$$) {
$$$
}'
ast-grep --pattern 'func OpenVersionDB($$$) {
$$$
}'
Length of output: 124
Script:
#!/bin/bash
# Let's try with ripgrep to find these function definitions
rg "func OpenVersionDBForReadOnly|func OpenVersionDB" -A 10
Length of output: 2306
versiondb/tsrocksdb/store.go (4)
42-44: LGTM! Well-documented field addition.
The skipVersionZero field is appropriately documented with a reference to the related issue.
65-67: LGTM! Clean setter implementation.
The method correctly uses a pointer receiver for mutating the struct field.
235-246: LGTM! Well-documented fix functionality.
The FixData method is well-documented with clear references to the issue and usage instructions.
97-113:
Fix resource leak in GetAtVersionSlice.
The ReadOptions created by newTSReadOptions is not being destroyed, which can lead to memory leaks.
Apply this diff to fix the issue:
func (s Store) GetAtVersionSlice(storeKey string, key []byte, version *int64) (*grocksdb.Slice, error) {
+ readOpts := newTSReadOptions(version)
+ defer readOpts.Destroy()
value, ts, err := s.db.GetCFWithTS(
- newTSReadOptions(version),
+ readOpts,
s.cfHandle,
prependStoreKey(storeKey, key),
)Likely invalid or redundant comment.
Signed-off-by: yihuang <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (2)
versiondb/tsrocksdb/store.go (2)
297-327: Consider adding progress tracking for large datasets.The
loadWrongDatamethod could benefit from progress tracking when processing large datasets.Consider adding a progress callback:
-func (s Store) loadWrongData(storeName string) ([]KVPairWithTS, error) { +func (s Store) loadWrongData(storeName string, progress func(processed int)) ([]KVPairWithTS, error) { // ... existing code ... var pairs []KVPairWithTS + processed := 0 for ; iter.Valid(); iter.Next() { + processed++ + if progress != nil && processed%1000 == 0 { + progress(processed) + } // ... existing code ... } return pairs, nil }
307-310: Document the FIXME with more context.The FIXME comment needs more context about the issue and its implications.
Consider expanding the comment:
- // FIXME: https://github.com/crypto-org-chain/cronos/issues/1689 + // FIXME: Skip non-zero timestamps for now. This might need to be revisited + // for handling other timestamp-related issues. + // See: https://github.com/crypto-org-chain/cronos/issues/1689
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (3)
CHANGELOG.md(1 hunks)versiondb/tsrocksdb/iterator.go(4 hunks)versiondb/tsrocksdb/store.go(7 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- CHANGELOG.md
- versiondb/tsrocksdb/iterator.go
🔇 Additional comments (3)
versiondb/tsrocksdb/store.go (3)
42-44: LGTM! Well-documented field addition.
The skipVersionZero field is properly documented with a reference to the issue that explains its purpose.
235-246: LGTM! Well-structured data fixing implementation.
The FixData method is well-documented and properly handles multiple stores. The comment about calling this before SetSkipVersionZero(true) is crucial for correct usage.
97-113:
Fix potential resource leak in GetAtVersionSlice.
The ReadOptions created by newTSReadOptions is not being destroyed, which can lead to memory leaks.
Apply this diff to fix the issue:
func (s Store) GetAtVersionSlice(storeKey string, key []byte, version *int64) (*grocksdb.Slice, error) {
+ readOpts := newTSReadOptions(version)
+ defer readOpts.Destroy()
value, ts, err := s.db.GetCFWithTS(
- newTSReadOptions(version),
+ readOpts,
s.cfHandle,
prependStoreKey(storeKey, key),
)Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (3)
versiondb/tsrocksdb/store.go (3)
297-327: Consider adding progress tracking for large datasetsThe
loadWrongDatamethod processes all data at once without any progress indication, which could be problematic for large datasets.Consider adding a progress callback parameter to provide feedback during processing:
-func (s Store) loadWrongData(storeName string) ([]KVPairWithTS, error) { +func (s Store) loadWrongData(storeName string, progress func(processed int)) ([]KVPairWithTS, error) { var version int64 iter, err := s.IteratorAtVersion(storeName, nil, nil, &version) if err != nil { return nil, err } defer iter.Close() var pairs []KVPairWithTS + var count int for ; iter.Valid(); iter.Next() { + count++ + if progress != nil && count%1000 == 0 { + progress(count) + } // ... existing code ... } return pairs, nil }🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 301-302: versiondb/tsrocksdb/store.go#L301-L302
Added lines #L301 - L302 were not covered by tests
[warning] 308-309: versiondb/tsrocksdb/store.go#L308-L309
Added lines #L308 - L309 were not covered by tests
[warning] 314-315: versiondb/tsrocksdb/store.go#L314-L315
Added lines #L314 - L315 were not covered by tests
307-310: Address FIXME comment with a more descriptive explanationThe FIXME comment references an issue without explaining the rationale for skipping non-zero timestamps.
Consider adding more context to the comment:
- // FIXME: https://github.com/crypto-org-chain/cronos/issues/1689 + // Skip non-zero timestamps as they are correctly stored + // Related to timestamp handling improvements tracked in: + // https://github.com/crypto-org-chain/cronos/issues/1689🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 308-309: versiondb/tsrocksdb/store.go#L308-L309
Added lines #L308 - L309 were not covered by tests
235-246: Add logging for better operational visibilityThe
FixDatamethod lacks logging to track its progress and success/failure status.Consider adding structured logging to help with monitoring and debugging:
func (s Store) FixData(storeNames []string, dryRun bool) error { + log.Info("Starting data fix operation", "stores", len(storeNames), "dryRun", dryRun) for _, storeName := range storeNames { + log.Debug("Processing store", "name", storeName) if err := s.fixDataStore(storeName, dryRun); err != nil { + log.Error("Failed to fix store", "name", storeName, "error", err) return err } + log.Debug("Successfully processed store", "name", storeName) } + log.Info("Data fix operation completed successfully") return s.Flush() }🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 241-242: versiondb/tsrocksdb/store.go#L241-L242
Added lines #L241 - L242 were not covered by tests
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
versiondb/tsrocksdb/store.go(6 hunks)
🧰 Additional context used
🪛 GitHub Check: codecov/patch
versiondb/tsrocksdb/store.go
[warning] 103-104: versiondb/tsrocksdb/store.go#L103-L104
Added lines #L103 - L104 were not covered by tests
[warning] 241-242: versiondb/tsrocksdb/store.go#L241-L242
Added lines #L241 - L242 were not covered by tests
[warning] 252-253: versiondb/tsrocksdb/store.go#L252-L253
Added lines #L252 - L253 were not covered by tests
[warning] 267-268: versiondb/tsrocksdb/store.go#L267-L268
Added lines #L267 - L268 were not covered by tests
[warning] 274-274: versiondb/tsrocksdb/store.go#L274
Added line #L274 was not covered by tests
[warning] 278-278: versiondb/tsrocksdb/store.go#L278
Added line #L278 was not covered by tests
[warning] 288-288: versiondb/tsrocksdb/store.go#L288
Added line #L288 was not covered by tests
[warning] 301-302: versiondb/tsrocksdb/store.go#L301-L302
Added lines #L301 - L302 were not covered by tests
[warning] 308-309: versiondb/tsrocksdb/store.go#L308-L309
Added lines #L308 - L309 were not covered by tests
[warning] 314-315: versiondb/tsrocksdb/store.go#L314-L315
Added lines #L314 - L315 were not covered by tests
🔇 Additional comments (3)
versiondb/tsrocksdb/store.go (3)
42-44: LGTM: Clear documentation of skipVersionZero field
The field is well-documented with a reference to the relevant issue (#1683).
97-113:
Fix potential resource leak in GetAtVersionSlice
The ReadOptions created by newTSReadOptions is not being destroyed, which could lead to memory leaks.
Apply this diff to fix the issue:
func (s Store) GetAtVersionSlice(storeKey string, key []byte, version *int64) (*grocksdb.Slice, error) {
+ readOpts := newTSReadOptions(version)
+ defer readOpts.Destroy()
value, ts, err := s.db.GetCFWithTS(
- newTSReadOptions(version),
+ readOpts,
s.cfHandle,
prependStoreKey(storeKey, key),
)Likely invalid or redundant comment.
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 103-104: versiondb/tsrocksdb/store.go#L103-L104
Added lines #L103 - L104 were not covered by tests
264-275: Add error handling for nil oldValue
The code doesn't explicitly handle the case where oldValue is nil.
Let's verify if GetCF can return nil without an error:
🧰 Tools
🪛 GitHub Check: codecov/patch
[warning] 267-268: versiondb/tsrocksdb/store.go#L267-L268
Added lines #L267 - L268 were not covered by tests
[warning] 274-274: versiondb/tsrocksdb/store.go#L274
Added line #L274 was not covered by tests
…) (#1686) * Problem: nixpkgs-fmt is deprecated (backport: #1677) Solution: - switch to nixfmt-rfc-style * Problem: no command to fix corrupted data in versiondb (backport: #1685) Closes: #1683 Solution: - add fix command to fix corrupted data in versiondb rename support SkipVersionZero support SkipVersionZero cleanup cleanup cleanup fix test cleanup destroy log store name fix data manually cli Update versiondb/client/fixdata.go Signed-off-by: yihuang <[email protected]> Update versiondb/client/fixdata.go Signed-off-by: yihuang <[email protected]> rnemae Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> fix test don't return nil as empty slice add dryrun mode separete read from iteration add stores flag debug add timestamp api * update rocksdb * changelog * fix flag name conflict * validate timestamp * Update CHANGELOG.md Signed-off-by: yihuang <[email protected]> * skip non-zero version * flush after fix * only flush if not dry-run --------- Signed-off-by: yihuang <[email protected]>
related prs crypto-org-chain/cronos#1684 crypto-org-chain/cronos#1685 crypto-org-chain/cronos#1688 (not include versionDB.SetSkipVersionZero(true) in app.go) crypto-org-chain/cronos#1751 crypto-org-chain/cronos#1759 (notice not include cronos storeloader and upgrades) crypto-org-chain/cronos#1780
related prs crypto-org-chain/cronos#1684 crypto-org-chain/cronos#1685 crypto-org-chain/cronos#1688 (not include versionDB.SetSkipVersionZero(true) in app.go) crypto-org-chain/cronos#1751 crypto-org-chain/cronos#1759 (notice not include cronos storeloader and upgrades) crypto-org-chain/cronos#1780
related prs crypto-org-chain/cronos#1684 crypto-org-chain/cronos#1685 crypto-org-chain/cronos#1688 (not include versionDB.SetSkipVersionZero(true) in app.go) crypto-org-chain/cronos#1751 crypto-org-chain/cronos#1759 (notice not include cronos storeloader and upgrades)
update memival,store,versondb related prs crypto-org-chain/cronos#1684 crypto-org-chain/cronos#1685 crypto-org-chain/cronos#1688 (Do not include versionDB.SetSkipVersionZero(true) in app.go) crypto-org-chain/cronos#1751 crypto-org-chain/cronos#1759 (note: Do not include cronos storeloader and upgrades) note: I don't use this. crypto-org-chain/cronos#1780 because it update rocksdbs from version 9.2.1 to 9.11.2
2 participants
Closes: #1683
Solution:
👮🏻👮🏻👮🏻 !!!! REFERENCE THE PROBLEM YOUR ARE SOLVING IN THE PR TITLE AND DESCRIBE YOUR SOLUTION HERE !!!! DO NOT FORGET !!!! 👮🏻👮🏻👮🏻
PR Checklist:
make)make test)go fmt)golangci-lint run)go list -json -m all | nancy sleuth)Thank you for your code, it's appreciated! :)
Summary by CodeRabbit
Release Notes
Bug Fixes
Improvements
Tests