SIGSEGV from eth/downloader/downloader.go

[pepa65]

Geth version: 1.10.2-stable, go1.16
OS & Version: Linux Mint 20 Ulyana
Commit hash : 97d11b0
Date: 20210408

It had been running for a number of days, but now it has crashed 3 times in the same day.
Run with: --datadir=/data/Geth --cache=512 --lightkdf --ws
Output:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5455c4]

goroutine 54258 [running]:
math/big.(*Int).Cmp(0xc00bacb360, 0x0, 0x67f5d5406ab810c0)
        math/big/int.go:328 +0x44
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc00000c3c0, 0x98fcd9, 0xc00bacb360, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1586 +0x126f
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0x140aba0, 0xc00842b260)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:563 +0x3f
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc00000c3c0, 0xc00d17b740, 0xc00cd6bc60)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:584 +0x5b
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:584 +0xaf

[ligi]

Can you provide a bit more log?

[pepa65]

Do you mean from before the crash? Because this is all there is after. OK, this it from the last run:

INFO [04-22|01:20:46.834] Imported new block receipts              count=63   elapsed=1.692s      number=10049429 hash="3d47db…9f2002" age=11mo2w13
h size=2.70MiB                                                                                                                                     
INFO [04-22|01:20:49.886] Imported new block receipts              count=10   elapsed=2.957s      number=10049439 hash="ff0c1d…2d8f40" age=11mo2w13h size=317.10KiB                                                         
INFO [04-22|01:20:50.552] Imported new state entries               count=536  elapsed=1.959ms     processed=16174976 pending=3409 trieretry=0   cod
eretry=0 duplicate=0 unexpected=0                                                                                                                  
WARN [04-22|01:20:50.552] Rewinding blockchain                     target=10071461                                                                 
ERROR[04-22|01:20:50.554] Ethereum peer removal failed             peer=165de667 err="peer not registered"                                         
INFO [04-22|01:20:51.168] Initializing state bloom                 items=6262162 errorrate=0.000 elapsed=22m22.209s                                
ERROR[04-22|01:20:51.575] Invalid block total difficulty RLP       hash="d4e567…cb8fa3" err="rlp: value size exceeds available input length"       
INFO [04-22|01:20:51.575] Loaded most recent local header          number=10071461 hash="ac386e…bab304" td=15436625923999431977999 age=11mo1w4d    
INFO [04-22|01:20:51.576] Loaded most recent local full block      number=0        hash="d4e567…cb8fa3" td=<nil>                   age=52y2w4d     
INFO [04-22|01:20:51.576] Loaded most recent local fast block      number=10049439 hash="ff0c1d…2d8f40" td=15386432373328820481508 age=11mo2w13h   INFO [04-22|01:20:51.576] Loaded last fast-sync pivot marker       number=12285044                                                                 
WARN [04-22|01:20:51.576] Rolled back chain segment                header=10078048->10071461 fast=10049439->10049439 block=0->0 reason=nil         WARN [04-22|01:20:51.576] Synchronisation failed, dropping peer    peer=165de667f354e430533b7cc3222c67e8a42306e5a33bb9a83ac07531aa7e23bc err="actio
n from bad peer ignored: header request timed out"                                                                                                 
ERROR[04-22|01:20:51.576] Ethereum peer removal failed             peer=165de667 err="peer not registered"                                         
WARN [04-22|01:20:57.724] Synchronisation failed, dropping peer    peer=2c02475b870b77ca394addba3e85981877d66122c40485ced607beb327d2310e err=timeou
t             
ERROR[04-22|01:20:57.724] Ethereum peer removal failed             peer=2c02475b err="peer not registered"                                         
INFO [04-22|01:20:59.173] Initializing state bloom                 items=6473403 errorrate=0.000 elapsed=22m30.214s                                
INFO [04-22|01:21:07.191] Initializing state bloom                 items=6502797 errorrate=0.000 elapsed=22m38.232s                                
INFO [04-22|01:21:15.191] Initializing state bloom                 items=6504447 errorrate=0.000 elapsed=22m46.232s                                
WARN [04-22|01:21:18.404] Stalling state sync, dropping peer       peer=2cd761db1e588e7c07797338562a35ef1a86b666721b7c8c191363d9d0c43c26           
WARN [04-22|01:21:18.404] Stalling state sync, dropping peer       peer=36e6eb1d979752ca01f41b733ffbf668f65a8d481c04d52b7bd85cf7c0e08529           
ERROR[04-22|01:21:18.404] Ethereum peer removal failed             peer=2cd761db err="peer not registered"                                         
INFO [04-22|01:21:18.404] Imported new state entries               count=1    elapsed="7.699µs"   processed=16174977 pending=17   trieretry=0   cod
eretry=0 duplicate=0 unexpected=0                                                                                                                  
ERROR[04-22|01:21:18.404] Ethereum peer removal failed             peer=36e6eb1d err="peer not registered"      
WARN [04-22|01:21:18.404] Synchronisation failed, dropping peer    peer=36e6eb1d979752ca01f41b733ffbf668f65a8d481c04d52b7bd85cf7c0e08529 err="retrieved hash chain is invalid: syncing canceled (requested)"
ERROR[04-22|01:21:18.405] Ethereum peer removal failed             peer=36e6eb1d err="peer not registered"
INFO [04-22|01:21:23.406] Initializing state bloom                 items=6506479 errorrate=0.000 elapsed=22m54.447s
WARN [04-22|01:21:23.748] Synchronisation failed, dropping peer    peer=2689d8adb6a1e8733d62aa88089bf30b843db565e08ec0ce31f78e4431b18929 err="action from bad peer ignored: returned headers 192 != requested 2"
ERROR[04-22|01:21:23.749] Ethereum peer removal failed             peer=2689d8ad err="peer not registered"
INFO [04-22|01:21:32.140] Initializing state bloom                 items=6508524 errorrate=0.000 elapsed=23m3.180s
INFO [04-22|01:21:40.173] Initializing state bloom                 items=6510768 errorrate=0.000 elapsed=23m11.214s
INFO [04-22|01:21:48.398] Initializing state bloom                 items=6512485 errorrate=0.000 elapsed=23m19.439s
INFO [04-22|01:21:56.271] Downloader queue stats                   receiptTasks=0     blockTasks=0     itemSize=87.72KiB  throttle=748
ERROR[04-22|01:21:56.271] Invalid block total difficulty RLP       hash="d4e567…cb8fa3" err="rlp: value size exceeds available input length"
ERROR[04-22|01:21:56.271] Ethereum peer removal failed             peer=9ac8df96 err="peer not registered"
INFO [04-22|01:21:56.437] Initializing state bloom                 items=6513984 errorrate=0.000 elapsed=23m27.478s
INFO [04-22|01:21:56.460] Imported new state entries               count=35   elapsed="6.076µs"   processed=16175012 pending=523  trieretry=0   coderetry=0 duplicate=0 unexpected=0
WARN [04-22|01:21:57.324] Synchronisation failed, dropping peer    peer=9ac8df96e49bce26338dadc6b0d7d03473441f2a349563763bd73a241c640a34 err="action from bad peer ignored: header request timed out"
ERROR[04-22|01:21:57.324] Ethereum peer removal failed             peer=9ac8df96 err="peer not registered"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5455c4]

goroutine 30843 [running]:
math/big.(*Int).Cmp(0xc0323a1760, 0x0, 0x67f5d5406ab810c0)
        math/big/int.go:328 +0x44
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).processHeaders(0xc00000c3c0, 0x9957a0, 0xc0323a1760, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:1586 +0x126f
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer.func6(0x1, 0xc0214309c0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:563 +0x3f
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync.func1(0xc00000c3c0, 0xc039310ae0, 0xc03902b020)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:584 +0x5b
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).spawnSync
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:584 +0xaf

[holiman]

Interesting:

INFO [04-22|01:20:51.576] Loaded most recent local full block      number=0        hash="d4e567…cb8fa3" td=<nil>                   age=52y2w4d     

So we somehow have the genesis in there, but the genesis has nil for td?

[holiman]

This is what one gets starting a clean geth from scratch:

INFO [04-22|13:15:07.907] Loaded most recent local full block      number=0 hash=d4e567..cb8fa3 td=17,179,869,184 age=52y2w5d

So something, somwhere, deleted the td entry for genesis.
@pepa65 thanks for providing logs, that helps a lot!

[pepa65]

So what should I do? Ditch all 130GB and start again, or is there something else I can do?

[fjl]

@pepa65 it seems your node hit a bug in go-ethereum core, and we find it valuable for debugging. if you just want to get on with life now, you should resync the node. unfortunately the problem cannot be fixed immediately.

[pepa65]

I've restarted it one more time without --cache=512 --lightkdf and it's not immediately crashing...

[pepa65]

we find it valuable for debugging

Do you have enough information to look into things??

[pepa65]

Now running with: geth --datadir=/data/Geth --cache=512 --ws and still going strong. I suspect the problem is in the code relating to --lightkdf.