`snap` sync in `path` mode not working.

[greg7mdp]

System information

Geth version: geth version: geth version 1.14.7-stable-aa55f5ea
CL client & version: Lighthouse v5.2.1-9e12c21
OS & Version: #117~20.04.1-Ubuntu SMP Tue Apr 30 10:35:57 UTC 2024
Commit hash : (if develop)

Expected behaviour

Should be able to sync from scratch.

Actual behaviour

I tried to sync from scratch 3 times, after blowing the database every time using:
sudo geth --datadir /var/lib/goethereum removedb

First time keeping the ancient, next two times blowing everything.

System seems fine. using nvme ssd (Samsung 970 Plus 2TB) with no issues reported with smartctl -a.

Steps to reproduce the behaviour

Use removedb to blow database and run geth with:

geth --mainnet --http --http.addr 0.0.0.0 --datadir /var/lib/goethereum --authrpc.jwtsecret /var/lib/lighthouse/secret/jwt.hex --cache 10704 --maxpeers 35 --metrics --pprof --pprof.addr 0.0.0.0

Note

Previously I used hash mode, but I starting having errors, so I decided to resync in path mode.
On my first resync attempt, I specified path mode explicitly, but then I removed it since it is the default with geth 1.14.

Partial log (seems to repeat the following):

Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.758] Unexpected trienode heal packet          peer=d32e6ab2 reqid=7,062,910,621,028,475,407
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.774] Unexpected trienode heal packet          peer=141f8901 reqid=5,921,614,888,348,237,967
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.795] Unexpected trienode heal packet          peer=0ebdc58c reqid=4,864,759,534,782,202,732
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.880] Unexpected trienode heal packet          peer=edcdb9d9 reqid=2,192,217,697,858,156,755
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.921] Unexpected trienode heal packet          peer=1ff4414c reqid=6,030,783,751,269,227,388
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.959] Unexpected trienode heal packet          peer=55522f26 reqid=6,072,072,301,071,529,713
Jul 13 09:03:57 gp-nuc-staker geth[128070]: WARN [07-13|09:03:57.972] Unexpected trienode heal packet          peer=d6161cca reqid=2,271,209,749,937,513,820
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.085] Unexpected trienode heal packet          peer=0f098f11 reqid=1,138,651,193,010,756,505
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.141] Unexpected trienode heal packet          peer=936db72d reqid=4,437,056,338,323,220,388
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.142] Unexpected trienode heal packet          peer=c966be1b reqid=7,099,670,709,689,758,184
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.187] Unexpected trienode heal packet          peer=d6469ba0 reqid=3,762,804,802,417,240,180
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.246] Unexpected trienode heal packet          peer=38d7692b reqid=7,502,169,550,455,053,468
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.451] Unexpected trienode heal packet          peer=84fabfcb reqid=7,547,774,593,759,638,446
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.455] Unexpected trienode heal packet          peer=48382872 reqid=8,242,958,486,812,907,649
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.582] Unexpected trienode heal packet          peer=8d6c20c7 reqid=3,239,138,954,101,181,394
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.817] Unexpected trienode heal packet          peer=4dca7c2d reqid=1,258,313,712,087,304,831
Jul 13 09:03:58 gp-nuc-staker geth[128070]: WARN [07-13|09:03:58.962] Unexpected trienode heal packet          peer=93abd245 reqid=247,484,856,798,019,426
Jul 13 09:03:59 gp-nuc-staker geth[128070]: WARN [07-13|09:03:59.164] Unexpected trienode heal packet          peer=fe409484 reqid=1,940,029,955,041,506,237
Jul 13 09:03:59 gp-nuc-staker geth[128070]: WARN [07-13|09:03:59.761] Unexpected trienode heal packet          peer=676aa5ea reqid=7,948,375,599,722,888,431
Jul 13 09:03:59 gp-nuc-staker geth[128070]: WARN [07-13|09:03:59.818] Unexpected trienode heal packet          peer=6b36f791 reqid=8,237,650,123,766,752,191
Jul 13 09:04:01 gp-nuc-staker geth[128070]: ERROR[07-13|09:04:01.080] Reject duplicated disable operation
Jul 13 09:04:01 gp-nuc-staker geth[128070]: WARN [07-13|09:04:01.083] Rewinding blockchain to block            target=17,236,615
Jul 13 09:04:01 gp-nuc-staker geth[128070]: INFO [07-13|09:04:01.179] Forkchoice requested sync to new head    number=20,297,763 hash=5a9e28..601654 finalized=20,297,671
Jul 13 09:04:01 gp-nuc-staker geth[128070]: WARN [07-13|09:04:01.624] Unexpected trienode heal packet          peer=0ad80bd5 reqid=6,054,871,301,206,217,589
Jul 13 09:04:06 gp-nuc-staker geth[128070]: WARN [07-13|09:04:06.589] Unexpected trienode heal packet          peer=507c0ba1 reqid=728,961,722,073,398,230
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Loaded most recent local header          number=17,236,615 hash=d45fec..85641c td=58,750,003,716,598,352,816,469 age=1y2mo1w
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Loaded most recent local block           number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=55y3mo3w
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Loaded most recent local snap block      number=17,236,615 hash=d45fec..85641c td=58,750,003,716,598,352,816,469 age=1y2mo1w
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Loaded last snap-sync pivot marker       number=20,297,699
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Truncated excess ancient chain segment   oldhead=17,236,616 newhead=17,236,615
Jul 13 09:04:09 gp-nuc-staker geth[128070]: ERROR[07-13|09:04:09.189] Failed to reset txpool state             err="missing trie node d7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544 (path ) state 0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544 is not available"
Jul 13 09:04:09 gp-nuc-staker geth[128070]: ERROR[07-13|09:04:09.189] Failed to reset blobpool state           err="missing trie node d7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544 (path ) state 0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544 is not available"
Jul 13 09:04:09 gp-nuc-staker geth[128070]: INFO [07-13|09:04:09.189] Syncing: state healing in progress       accounts=171,363@9.40MiB      slots=370,618@27.58MiB        codes=158@1.12MiB       nodes=2,200,693@656.42MiB pending=1
Jul 13 09:04:09 gp-nuc-staker geth[128070]: WARN [07-13|09:04:09.827] Invalid header encountered               number=17,243,765 hash=c4b4d5..d4945e parent=92b4d0..33efde err="invalid baseFee: have 22538549297790339279734171310165035945408590406000924764119485189068901105624426301855089982168153411550498537209217831373027898239413607943974690469634015283818785048852579179651325625259832894695691515525064791337815595940419550393589714109246997359284111805124879034697899525295424341491246035168509628172907509521129100362692207788162713961236095039939262974933566853686299190485776674002268522718733250330593316091256945728471735413084714797754924980549384657782392683361214171005136784521456234054143342537901317065719284303058998029664626932578749925783310255654602479250539161990713166752302906809618288152821109898175669726923372552932536418724458948591326277142444161014876627458004662477650951029823838076828953776090732435403021715630746282681766320038663990963863852006267626214767200889504219768527718466717599099673892659859747365971796601170232460197088684606947870095176023335886794457625217602318152542483681965612179634372317826117336376706664089136712108120539547072718335031282627884301573763383449753105875635176913668461331617296072190938190526500434189888042032668278466203146748502295460457919420557961151890312586941301685218753984111290634306131555279265117573532777913708143062960102358953157436884943568174397316890473488245341580691877538876333527638155271626960537872257996469475505696009811210694279688449941315777979150651683299594288562164573620993270979364061657267653116985337506911004796992217772550036279419192875689594008383627196837462401804415246919936897110242353211888958661242023396784607186863132104101093978127746661391508350906343553995523079838337620692354697754370659373286798752309446844868917142884687732336736106214013904046225386595485661424895532510789607493673002702294575915846343976854757147949897233351135099562589593960168076464541062114481669309255008626350861489862968951017692232832369610770811526170395193613236954597927339198555463805832396877366460174937946550299762393528183504184829255548931568247797978797698122775750837402059968504890632969755949713801874510659189797681534203834724833542922628924172096649715714933133664760700349802232640176378784747313415596869345694405077811290322872837740144797064793869828958750188797336611549304802973970815744243072656692321830436116720560350562611392607112503995726393941746852893742767968203856460429444405513620304709782148486901400985839227050371933464865642262684847203032680591231981971790053580359745217196362027067715245263969569264816750521399977049809850423836008190534031242180390126993908121391345920351855849672111288942046519830185399162269823142357490591601940496845313896652130813920258398024023864948279013652115209210524184901118105943910519657495808408974705677668889667324737912578343789547619101314453291528004409239157496746318282841982409899622307774904465599023316587147121347226756095841106292951889112374026667957519790443674340532222718581359676004068762525633351700751657061154148314381661758789258835487117136795585595442846800971509309285376590095051444389244418646637476206829622395612729248627562852550708573501632504536658692339721478231465835258703243602227956263104651808811557823793675559266493433031125107420711670304328626278768351106518166557460756077753012934399879810313615911564602930969408935562135392975365833594355072251943553877536528002820602112874398776819430612454097678199901195300499686226872833230564100265381898087034060865955603133024459604314075698003210185351116392881661247932596571948983865799057110238885927353070404513705621908169202947435378822318135735395541772183011209182979340159966101493021975253406651362012295623277622220637911529973882505507491600241970011671105101863576791614706247352366749420756300031579892891800803202128875032061150202897249022414322415792542142132123645705447488557348565012959663775943123272534092697796396730954751952819158937486841446217937796604185991305586482736895794390116493190982628951737798611247576602623517798182227534992536532884926412286889305812423218512846017869970608451529611101277040498093100250948795502654731359139541931018992055031180434158212550193554128097627346699531252111573201395249643466759104980299108950799853927502712942301472125217333447539171249405406154409779006906066767229907588282374638056288934943218495483030683245112081221644756545771856379535451123182694050771235633545583838683046677019114459013754336940937623309785970236084554815111321263295212309725357798156819477692545504976104651526167637649933334657280776159544715326286751840556053774656056250527231403759976888063734388633729988591882948843180969262720464923139975108702710748435947831824654937585838947578122885472741518582292155415480494642379882985920042731354615491505736694291445338432971597711041710693229427704368511526176051145418505610842435081983864072098349745753618841101886078954234010627116528579626158904530215517557455851958406991465039494116985781078897176745948035605111414940577428877851770159805516510167355389325051404440938015558666774899936640456780949170159139924798181017614446901266602573962832382628356502433165709009069524593612175960383639330735471289819000495156825673148828729717439543236554555375029265477580259514206254601206896661502603682039131634740032680040486096665578813216794654362423796808641893231822788600034277588015129665964440572374331617367504335874113054596452484687908059558432681308848248641647339373675604765875284160683589661240529819527885625206402452885307419551084895344114690416027507411612978060775501613948884395999224922726215834543343084188147852114463871770134480714628043841189983625840043915978444481084336657298061135385263569903458810612349253486579664786598310718132148704849393200096712371791354310642662858980065383365000887200423253498297480104472193039990270849406580083580071027733995559899530259554886674946120813857872865756488785961541765659086567324705100129339640686412394367089160335144425346493190544323575752345741477096327845067676741680730581763937970870359316396659007877992474508203634620112774713943502323685431497396359039141087576614250634078577427838915796955509799601406035266013743180122717744959431016608976898726171450706352259672017935243685609388788398073287123132263157978072451214784777966722604861993208864636519533186342144099068469264181341911612342890535308059055144696465460935796920888088018381253171642294343904559331864934282141399384498963704752951189768259253751404279594240075412589273935231603966181701361603039453111217624544139621971072919010061898493056251074652695999945971741157358537326478320214069114952125721341271647936200938242519759840937168141657575299772081229640863674505748087518690425045505562509503298198893955464221317136303930191096192783195254085355732467681978969033417553358870683956487994693988502719153683956024684576269462580078763136591908678537669917807760807589054160911904096188134960796080069684999193723081282378887533181241637759794565448230440934941914803178038665709175590382920562763471279293562004967000198054368950259272285038666321063746832140881719852270857160093311365185589523645330207227400400810945242577183183807149033888542060056133956803922485194160898354198360853648506422386350798916611601377078730503623738596460780379219692971096196234254184332278116480901741008018013774304287181715291000848050261584399167788825095626389571230627628545546406827376827516696776759983729838763600093595313390310119609454116065822554454039090736993414632531919403169344983848924012266507871323222919639880352984425272340675212207736635759022798742627069031323696685095645419088378153263505273864800626560286985482483828031758934547670669492557872769536491224542865437296816542426797430571752859142158551356704317837010781226845425497668115405682284960675788509615496006657389386677671624203580368354009835774684159697208462048187868293876668325937737563141561060766857669780696838610193537628735620012256938268504746812212228521608028112489603037054595237804583645279899221553272365021981877064408570923212135981799119017738098513291430233633187485435445483352391356084355661378855589920459973955254443416139846013687460508535417565238866309160643199019682037892493640916775238827674489760838020166416209951564359217865909318972910644073135833358951946945964179745874104460376118918078909109704477943526967049900849213459464054454446647004903034456499945945552085199776150756221088677015113227766481743408310553049528747323284923225807577722663289819576905202520484718599405380485302711039216266722062844179049177805320752386002855781295241696066909919781828359061372943099760115135605171419599639104985220642203337552416286184170129155427282051634483214038001914197384107119894019533332213883754568411852429071763015175524100517363110473558675059436875634170987102210784565091685517745297504484573553434013166451929621317858511865370529365610450317290970061740208313474977153173808865598272648188688309934010440095153142525483435129775101873799554572182533368296650441998163297761203935013001743050718323108708507729260220297573852947479432742862711824287944684547185469275781287516176468719867974162954924645299850469732880716039729660317821393654130833427128645065571166313044867461001169432213629025809630317351847400927088838363292642271448549705858721761403015675598662076782037617787670265267049759629561618540362765517033097560498607976483732948263957978038489014408163681914858338127416898483506533562244646787622119108230323851490956288387370740264471020284092737309937762712394342900078140873155620747478277541757824910511519635107524240679739178587912507666508865536637196721870375000457682877247942935190605546934192815996876024225981138363104922752138160235552317146168300647262202794153067382766487760590620186475961425910454838854496649711376617219958839654846270960150193146144215686164639790443339053458547398610052752809870728403365260885566253024551499563901330984196385617546026127007839934045208161187654165172451450623260957644353144772299504956773845119656386478972479529664951924351490994287098253538375405759898981774858283236421697224267672086217310734087686582395261571201721204740467642110908005644185802109982932288187382045694635607071172963828687080086511862466859926294055761105232483513423639434510371791850804290728033909853343551767360009983470285113131641320228020860830231472478332844212270656876014684750120238287667637076583100314825598242905561140050102148504003359545748849876872572165065136804160919757760886670344882257258293451847445330803998104682008746644592026687750575506582234425838289249019295770993281878252160339462546220514623497902548613635021185750095020896371732023005020136427157157277706878043819275715379566599047960923073323648206223618822828069180800708728891853706741542070050127020476954637258971954365346576289978381965777076715297116599055282683345549502892862982068875667803120194808174275103244278660640203695523896640682266405958090565636293946266160173395282276345177636419481007250763085620777603507940719279076367811541083258789582050452474394442116093432009558195095864630749067306479193897117918577873573965555856991577383023566939006433413006110388702546519644183577248876780680534258879506936645158542033055941207626623638729891261255765658703387137559817966669787563379590699328733218785408605544881086212963164436065292485114283282722042733326637470137796676754931325916034721988264685374924748346193979201404489929679342234812333883786650771556303211614919314194117210245130540382708281304125209496379699872060534831323227716378730782170988527019058061326022776294664699164142237922854738525089961116345364658991074224236460959059755110309470872422733146636931646884118308437456589194321631401802941631875260016396944991740104702836351536976774126466142957448198449032508315873354002943938770247645221386250480190776733415228773702340956282207676188203960506677598698079949230627457007219801018585820129488809136247292654209319437899990024569869912434455787304670542502736427583055555681863030344310750864281402485917705828515227896732209313304087804291336220377701620215039816210501614289538449817862093043551072878620288692646067443702070859160390143143733103936986742605322566976695594340150850770710065817295073081741013117017260468414616723451289824075052625791196985476598001791389384557621199487281335190084648763782679259011027062474049660317413162541902740729235106013075306209550931732584594091604606106336628819718382486430959853409413416348036335846638276992757538069278978866664626666531936433545030974401544795047806333726073801627740900954671279221351758633489189591012427996726776328201549581935196565701198478773067217809619798897928568619770445011387030511906634675357615164903001098694421016925336498039692011432329326703669465252077441472097084949987162542536887132874780239697254421388458198406926485983367776179426109261470993269908611366304666342699415577365512087507732756182993297573901090432049391423989269139536557005163620014176582278184839374992376541126848445239983635335588168148023194687889500537255266022647964092738414090818764513644082402703084443029700879171177796549314164584467129780409304675941220749695114253664556886757007323363912032577151917592242363550278986022173604345971127363452932138863807393507611124773803264031249614909125395308487019187287446090820736593591836123867946391559164212462570076382615779630599293042407059722104996465995090273803100713014684652180088788852841094841855547006285545135112073677039521565873982339695695742980828603523816364507943095340932261402463850726454016996547921631926647951198837494103219593162943115016440192437353873452151098315751616520138857971378270213114299727853035547873833008477952266467507703371252872030873287805699467828359885619803581436868405006691494059778438294203449183478176050090703085483304975015292700527716266148691492086151613839132802494869962557474683171860648422985867991531875860261385314291545123218368906274989932686373919353464762659655016962046512359716564667226488632914033161886631817912653913269862951908057681055781943879370702861470851931733915340692756036688215783086576770650293782651180782872683925038977967418881006410844153893956485142720921733367134355963132663605192298245791358765691964317141106241112974267597940562264760963669593474568647740857248607287171958356752741388300659584614909842113715105541603815305532473126341052003422476377175553605217471986107825056136506359396614776205005017013676530189175458521882406184485036967988832910112751326823548394249393238859940820585932163510405382545659550507775795961157456776364096861988353358978973554428189134281588603794101221655616845264035056533511262472757574432263331414385421592048242057867896006075450228883455179995393624090945564123697599798800447699113848302116767892141602110579138149167755302371429090736009980826053423284175193341917264192367586665538557712279213068873573415716538838781136916765996087936878201654168239748567267335144372005278089679073214970312741869144382250132828859183027774292060221582819032972789013355003068449562042619195517423064563284474048589392767526487691570265362975977670679159049764158869600678627826517305728605523296953883902703077589248509882009657460708028052972829174186860181533369658159933581477667602850617354426163320667037262211274030875693383535536537864298177176491529674675414520528887206978207035020719800419551318831207028589803968917539250324960147460172066636663234141262622204953432231547497022836386713676237777538259145401324245230930309813990391405981578436772445449551701418333379757144730035449419276631037890904290817957260562261030987304934745619485959697734490593356354054276519220173215867028642043271959625945343650137567176667132173721109608413045437454260569109621842138298670540639970710636982968871934454773211082958647150508295384767804908518731738030560871366612792820311204020967774686522851210875190039935872725408165439562429952367102903613381913635019358816758703815569189591517366457768096288940140495971449191462499550718717976856733386072415766009402492047185992777100854234798066600103053470531696148052425886359510573024391209096803890651930218124386751141770351988775487166352818848087212456285833983445855085853137903532085404649424599126569198950669284686907836727391071617995641481256922647449013864333009158050759807515420717101178291692981213071124554364553445789703281641603928058785805873475752595370727709186518218030668127100188359145066600059644422866488804970870967625643449690824995913702868792894921846891091101655748458539640589774746910526532198779727582415897730306586770298090317203913940658351738017709488726160382962211572444753121047164767720968106138210533703037945553742501316614106324724185493877288670976350583248543724986807455750531505661192684007367145113435999722197645021563079838839849392149679552985855836108049128786545796401197730379585040771382934371029959202018521774112348378726132381448376203552449028892530186534228720896537180664643511526900933993160991622552588484512243432256212825831277949732502040740768201324550891617009325196898811918771006141612223102541503169424552255745007959172200031775642288972959862932836410070894196184179576242551358422250095870281140449535719668836099058554065395451726820796352345540105141318770994602344139509996330002080083462952317931140108132471837833842617308742172817003248670884816005465407025614241886946447677744209602532211254726254810303637813460136852442796615805954958830072567729453977535223945349808392633163095857213002111746482014071238864745217192739102773426225277405829502623503052106954389578132123982105143123083888300827888213660230519771626152663780084248256786023985201956789598185991644767079777146827874536087257654974226999355374270071811886962130909716830072366803266692988769178853068486426463667535543828697141175401286750576783443583685835809280891741704241204916443717514652678749332554678136405098846028076375759655485267903470815448322708622304204676758750155020259521890745354267306426703364440400032477535979637847949630575165256464478405322300899648028204236141593024198445271656056109481089219245451289543688573499567605331314576274138013644757236950841233824566813408296807753110022528680681664306658257298042307506582456620218930128304491459440980657564994926268653386629638210152292164949205066126376279250309083355401174134175921580109425436887956517462893518235029633048849383976613085451458204860306773539860440865101833686077709510935654171976516332874829226370335748405797350265015530242026063481452270998142163733410330091188015629486022300287665686437126151988321987727803898440922881616354310229833606308881184806494959714710981790761294330926281462278350782032312759511645566323491082403202863946844999365276051847867917483586376806306637357499674398062386165202362049653194190642671326295560000511159024810359781773947775187523014890123127311686969933194542774298416033026104499680852505311969534690801129279317263275626244463226827734106140006907551762393907438470795512792666126248366442414526759957167333572517908781348808722013762042234755689143608719487607797221222690152493190637291731548433246295813515176128815878325418886417901064157065466911798517853187880029133422071569443984754403582118176542960659718279255883854907159187610678981091423803098446812925761619857842173023150670059209920432836302662991951246973638923530578980704858513783837887522868341022426802415257464341370430348861613690872565338707658092377419400507713317407966423057881684629684774397027542702447320722970969104506947447593737185309232581203974268132379558941758360048387039113245499416720031670364684060989377651674760209149378480441705472105506251432320247280260404469483373771127602384289556085157974223456333594569776742812330735624997332889267934412127852930831426999751031550689489661224833801334200917740103128971496812156874093259248012607871177524617084041312695695771603050685679768282272906355842422107474159655342860312025599788562947744256148555032238197300301595316992909497856200004639284380589826940203574321685796570524023850090491916699595863881336254166410315658307822924234083961809120599528826643269101461129000466588792461807574033270223012554199271587701916075732029199962728821627166676833110287289821524066498453409887404326412401465557547485720870961905547460396519134039564102866680434751165561743782551548738834543841538732333816368270517207231016717727739556943825083762852874968887780118509185962794512704686160145007605272779112149906511715721694702916310456761351094639751770603599244244669848612956170709047485725257947209793377869034557314662825734003961556120414118651068588606246428917101359054761244821133381340569078012929923750119234209055538332923942831649186107452535416789970246109768179230104738913155319321525099340638765078060842476945751767827851363485414610033219722998578822398082711369410059807288410737918135139601966976313003541894486404033040304612797209808166657814816640890561106849529570709414862868791922565134448028943567624552811661650668398278505402802196456101682571943314576702713494303358474583211610868320854376602599251297876459454704228156024205442820531229072572783259009908467299637908471841455845648162488000187654791733348053969755194446137995912806755352792193783594427670222512824856316967738394342164886793417085326209617218428162950623401162621129904245784081836506946175293592006818519833587777065702343197637874080432960135864844208086111480419872647546794641506338722305348470433422313112483173635980514235661454691007537213277877981149359671897227494248332154433484545870270863041705237748496786677969430016208681538724873145718220279598630649115087514987016708799405886381041471699511164389102831071317487630341273077283193645459429909316684884112829972820344924775402768032019142317372232918157563289015460408373587503239906942775260415567537195711582725352212388274657751820853826283198067944047459672523311560291310952294940581709892638895240806732495661494202209219272032216818741886764313891220342208620720162644725168629624901503392278028729497544810514298628620562090609563339907603508673242598087368828699065793809901858710051888832456029904869686044467164845953277942368933173007979071723696654615176350552158081413831208760092555825654985469980689533052154329236854832842035642991980214288091499780677151243384954756158585532608348381954361793807568623038728451179933885887277218016097200693335593981729625212894204565514685356863362129051847926164594607564129445015727196941231325775049708774285362257494772630444608428222298891930657582746439054287927512938561231488189618880361229762679206943858650024603971902428274285674094321061845072210420561770346555286094663606946736411862280000264299138326619425258151760358148854451459340549305993423551606923582791503987063338612142426435820474237909849091672862290427494789480366896467902533636228330545415438918664936450113391927052515516820432872600317205368296858860260852097500535214587072362814705442698366794258210247488360866850277315620868423763792958401176487375304979461009527385273858163042343412456226061681209549995078457317559407236413308715571942263540527970952447198479482469744759466468327651999945654726731361231194909978585895235354693148833199196069138575703812138652699917048927443164751130170217074017926636652836016669330454324385846532699015281130750527849786820131840216681889106019418152723275131741180380469725139448467166331073529443312589091843734659126225906550213514390985515382415700189864256563543143382308109173491945637428008209649306167623709516409284865006946759504345596296673367191238870255626125645012011251116474330249729724950122862057845507914838372107606951603480935265321388303979034362390212688615023075306138223515491409909768190488879749164390278328227827485825820737864041813975818073490562185284924719684762209990925421801394975166057049840866106850655572904462411116691853434247564685840094564368744502855045930207244520127114284329124563369739952107549977243254432012005486282533471606782694020270124038239606721094221637046470133583902871408515366695470754067759342257940550670009350142150632963344213949666658415156805862748720145471682527466162229698932576094495137543034585463551907149077139475630411617461545132561541022894653198710799264298592403321937236013311331003443596563916196213444662047411527040375041720772984630860233357262593222594161960151233380276577338467002473108991130696391983142940376299311151487448432066517071778621559266101508197629628012334453585437908666602271368320298660322118967967403237790321875916528542373129959902484932244992736889073267160579316161005643777163232799343031122269518646714354287403119909149723380180249016823627530857459954929389265473899494847443462017694427961770678716120681020868855843122535244122006073020664053099874434288070759566709738100543724846323270621385500674361260301802938900549733723359975845615469394377010698849334447102667827903212309916172735443332384641824601370435017562472833675643891179023495432079331699081877931650559417237686741664883067023362946215847810602825299023419624787572077899449607328136517839508909333123181139691568793109790053870055714639043359758470860139969948892138001836788814089750005393362010386188353393739976134797296669253991054943122421803642086000002734722174210409747530420918080581841941912666787845382393902683077613198740849355965301686923747566274452130388439349942776751018911992791933451372884425127632572434867189000251963121746683049816914440909882936285765373758037395968104103142886498640921377680016489895813359056877356186889400147348367916441764424001735277516154435996655933986950904297588470077527863350063782815246923527836408215514278969688339848715696853300579984888507274898030163199222646621512633496273561365257921707442912384601867533985647237341106650632929260005685018205686765116133315607086227605544261645613120760086843092366941473587621631152165903297112758378912293878275669489501203876627805537208143737139327929896863631315402317555414208119985666902349946538237832769733910330907064240210870383421990238001086831479088459722914769467692443954474685788679257231705414480603129134720241373137962376478622643276809779830157862618704472449266755747881398195549842074438251764405703055333898406465397308805565030895968939697539387280867858359721448712333016159075792585309167170627096875066723676725825099551561261700159368145211805057236293625790227800735537807886351296758064368857362979822455285500937590555078060278505677744290843755629331034613509477950393059557548770928716621336587991684507450449673371501869336969632439687254438679415819630514848584245777308577065740473977965660439153508437542955905854841337436829716944045811358831805513410466492590212572579472386826679338130142970101244379751772674527330943861703326989298111588586419591347760460961359563643690315596643977317559791532364827687741785065531518878446392807962238560466769252568364264877612588968408122708114168434747806426949976825567638958138757249352279612894391253327318557580506986960241728683213568566724631180237264047848649020607930430791007256770069913460084881179829310599469283613396827888548749336676617562527081139131078284413532042679018757572161226425424775103275723894319384568828736882363548101044909565494570680986235652683146757672012320506255999006907779246770180104810061461136781978704042635366697496985036708476069191852255001694934953237409996716776478949374552455761423106103791697409775459889406869427410893097984006305901360796712835720635495343840027421055947636481426326745032092644204007683679667728079302738904173806689404794226822842824448068328621371109333988124224578012306473047159027590557537990002747345596092083288113266551712774285423643381867949807877716031829874033460116832432294806815443248816323080714702929949680893455165288548835737469703000807502322209965496085290502868140753595732161255301608456069582286370364127637426159301112361192655425900859288501183196854622659963773024009259654328464114327299667635418578271748279156779508277603915772771063262008539243194257499252174646899515888370051081931060959885161886168583013055881003772713552348725978560587286914184861302635234843018990597926401518038560230173881351284274829252277126104238430947717254841623191022667356714128444332313818111444777478805156721138571581612084503474277721363766868577038209515077618356745064291670912826875664141764917092145924688390671645948453530293357416417569838364047880414411823244451380225698794964316318127505368556651285873397423969866449745946958423689676402971574961204815486785636200489503684893343464157650421417214812974232796630571439632113000858080117029866898837517047192894961203817873645967354468437205115414075918972512343122911111734852438610229332418669461967014957066143452196804922410474072701383971707440636806138856963662675427673444803470150096723019095691980500720743146345036883585183789779113054778307047933766733647239303534597790677211152515977147580568592565257698532479613529575482736474980240783412355187711179254826107344192677773241981157460865984759508261563825645532641095235137543721493612949699742408771065976027081482336082882316661519162720312172490425164616631309526009865508052912584864674442090694335847635253963546636917164503258366121536105434293552953587091154271316414713292792482310968268463439347395956603157504178828372615771642242741613146949666728593140973176668185737762089374836071967642776118049303372596814885293330230124764881512282484009062684050063331125666270360747024553451504116327953530010263586351805190010781850423744281521819360509112662642121595318277532544143911313897950616213167576838396542008908113340293280590222590173072690599306794023755953097678416887533883242769347128346339837552845456146619153276116281326028417067911007572417074225522784214672642613979254423409429288759845592114841435041861833872523341730013612000043760957763275429291638789708588077497571340074392507202969877449875019715123585799054553063001340865605237169398336154176775207270397413305336206007861464788771859264474580222963232315825993928870652676736518714153493396294613566777239034847925339578510010405706224747379728276182706802881484680688823375700279261387931366662882719439179195073424414981735391941452178036764598945161888708455864491137269059355063858737272758285824703117492449058416065921383173706690546633230726815209475145748302856642514671849233343961433181168749497619723263950826745789373201150641766343278383657064656100104958782338945397598291854666126394949299901987256664498675576487689566334338235393750018813224783982380302940269310286553036950452768624775312677354258362317980232480599655892601737971605214456689414625051143149583130430015777666932278413376366490444799481994730937670144444108706475876364867963715493055413139989009064332461144764401332913216114174835230286790328260273776855482704763567420189045069407995458463148790202419905616663365656255145301923609398584304583918834530019737490919615364187160782624001064879131472384015496287137598952224982310537785032007424936093489750862005252362603204138959347827955672274570807193086354620031722701875656998897737063594680668907646074012561293335044071573346709693735604568080452486460946535926076900625312900620799812534942557108232365892232921373309006833256770623478329419960641573465365285085930346317204591531337595194296258358931619650781611883034923888677066494130909123967531964639015465489399954805452250081399179580422177728395024063649584361180211424821672571510538673078243326894153426539296503386005913561633619244503280032749612994689847604046760098267840290277779632563725197765582344044468580510393108259060611626151721642021336650816442168451954816239219383099487465904449076674353377619878310333633264084662106546205663984365029944691649507940407106264014744868439601525014119208450339353813430803710878871445151854431531193239966846583532506895172996798365496868037054342157210707692083852869985652403632390916617647050093190636328271114557529896734110149888963440411456255869187808421240179852328052416007345531539555207094898569484845379959096599350094013425458676859043118116423603610266960361888952320659686828927137809867399132935379414412279057432193276025556820953224776831664615670725665696425549414420084849718150349250844799823559130874187536675245390009975985346003296443074111161948986828130608030605974878642867774677559871211173628529334970477879235465718384914491913151356216440732481872899194438635359954572597351561435884828561408726857951049768224226271609357650351778333671977966629668934989066861460185349422445432557309970272764873129529277781563035392740559945844320598372421088027214115916544693422762329960005549998672437920421535304781232151152039529540045010598348189026056025370971058181636568298149047666035395482194443221046488062640359270202086635748784221683969780711367699880454568235836730472782858533457553463574196742403481290557721606527522688935188791305882626070750510960863348052239733068953435691594646139078026813286096486291844251776902325749710117240903095985924242332530656222227250240585259686722760013101711050251625633714023906837923193350531448113294155286689967029409927740886823664317399652214479270178344408773382913862695533899521553611724396716126160628228518313740596163337171105415708101979140357785084231877131493434358970639796528290015640237905802395520627080259329157046098136785443473949668933910031509536363803774582888195285042902418188261325170993829432147584257896792560949246081531272359515471296910372130039801428392043046951563752050729189275897400521949463269388073629522792587020447520100725563446835140874705849997426089662261671694936948098865531366264377932080369572448260847687072809990085863797457844692924098639628384584656404624514870766397811490409087043656055980390832392484773016020668436187699794242111888328986273380851650263817152496285057132446203198983647282759391899532263830686606813314361403667442338908245262527772570962918784692397218046878895019438361303959406705382471599747901470743014338818298194551694866087821845114169580232888277089058428781404664752303978461351789947740783482130440023346471156745212169438390794970572061653831447416945231622998933830469624432379015382451072316333122314761918546817666641724975915713942441184842738858664264987444177949286755849393149124567487216895621181759985776391095308565589036151178335698523619105096871666261908553086777362285580950571213245745172850775424593197772037309984828385132236524248654499284645135738025243511590143079775750053536816186436146795382998161433845157614312882542240718604615892079290727879653588179591671175639247734428338407893641753780166923093328041112342517356866510773925150448037351070348511397459690808952247741970475917955685299996204560341798129211119162188990370045633736990609811052156594519112188114558864570074664536642410881392342494665592614134258903540188283305969030700654116962504469101721652881896770418057056062113257207982005921457992424494909931179424012536222512492587398106680314616895156479473160626553473687058365533605681862818437677643715337356463376921522910030762661313903438423593640887591734945420491761832756294738128223390145771673025896624009184335620502407008565177673536133896057390562808098750293930099539336949778636990362397778262487240590580039952801898811619948974711981232653251962315101502511381979681449423203989003387201575497206044583190747154225267734561961607743076363516161592206562734594375725614966966824955714068025627478144329338557905801231778155431262906085035852372596017582518599335172636301833475834366033060186075298770927810001346471372332694454405934569624535174487599439107084807817861427183086446162201093954527360687523384804540288890004516031696376053647353081079155165144196987761879441538741482783261642749116566239615300526453831195179869974214512260225433962993645735454352518782721548114849568195362503832113145325561684994063899775154407614663724460063999180806170606988587532089521596151124691658547673803669268791976322768588039976615327660191174113702075923769412287969027117984430316426254425700118110403637865678872891910544613427459812729261229511758671524758105774081108674357657041824366291972172844877556355797856466823012260001167005390625764099118818877905934827486205393319945396562591166131217217448288590329045176360136799640239318157723013719500129097594879100130472824334674194752801794016524784346115585715792583355829682250295904870376491981140473819049985219396444962401618855869957555407201699621118207508526795672329573485679416412381753875953769592906257749540015632941023867717422189242233744977715217067028116628844804723395383536933268909372943130543033137929166320847699517505481112272191965571337120337694948958586784617946739560764256797834996238658675348015919553367760709771821246879151053100775525951678185936789646137105194353168806531391247738226816351631612706726011189949319336032566860017633474840776935176897680182614820961596122465995226339910201952274946194020798563825704926641004458431525360824337677375429491777753531960564821769400845471377360988874763295227957864310578682727415824923976553746712797750878145552890825540377931440648312471540066867453984182923596957910747560411087058569224079633585317099143551118039254561463156930983023505299224398910177848871403561878744754385511336553421839997331835055606684982227935564532236701187091781793633455583742425035248630979229636231868467923052437898362505975739547632988654143165943296459320967698111024110111721088590928764506975415994314459328073495224996574978895033116460709876182709866490745155944560946583194369063903257481281311755714779754244502487704711168934729207629355588251570133955818043551265551492660949841270073390021541882183693510632332777473522125876353570648213844714310969031729010428404707725886351284419474110976600907991764108959796648706675972598967200081081704579959524656771713333559555360007164241205385980464807752090893118321681423903861857420630127061695353050158746343700136648026731603383121777621789643813842455954937880501254234180098040248190581426435097913336982622730611863437664484423599147997008308223137560655369017160720465545883832227771132753055798029561485388466701315186594723130769270354036524080275997234681481784319020849550557392658011543661211141193378556081014085776128053522875014362453397657628861202407037539035385178164063301989936056314751335154927262621644846336441703646083060784970859328590859954989442846931349720689361608846645262140260989157737684761098920659772590942328886762414911681715226706393255734474101562239756101747917240081502327977397071836922359181375537907052920039994287300541999049450409336334886225164681110496150394157030652285248572823761749674924001312870405629993031147364526138939550818104087955946166214815654495609133317889302978367721188368019326183634100590665867366051426829473180341575959205410727603593331847854156318316221431841125605847960824180514408231542270712988490313049025889147029924809567955171853889632150833851333349369485497795741078284011424514043801041495230853869748519466011558578505600463485694841159413394821195314995630570701408491362818313067536869704328600957968818875236246987190670814464009765478658340509386527755460268245387616823341589644670146312421238197365903986865284236971688666844549942550606457462580954009397258975671631222293330717405598436034115691963325428241761380022595676578028648456410766444866610975897030008156826071943687760173703527023235821497529151288274406269592691831238170880561675066910780639540336777387479130732789626421821444446358907014148673072099809001016460851950039177173002386685993883086798872374600398134403217638257601369269803262299418843687078269800161432358562328561178524096550257648669872469813209569659020206967782786349903894010856532275665237444376804616572987170628994700810549627023967089991077835590297799471797066735057426995306428035569928427111511947638268151852327388528107804446920457785118558951598127855302203300352459673386895899738648470366590272286619723986347160769391334576920407329103334001256878039704203676907929022506235600421997740508380269689527741574068824087529532719318029591573145692761216963196029036470727139941004003794438406712118233001741921674868242418595830748710622303009518972752914859917623042518701154367349709955573369811860157720680294591375224277935651022428897693377803137515272128489417292944730435448933377005334397710836129178709244960460251145476143758867093292439997564476382954270875945385266448073936243081692719243848012645075214316811213090924730048717706312639779369514512115277106071250610210938340772431729296529960336998572250307484371562958393958645280954372049431199916998188234226047248318895890805550280131527404708360820806121091326401091255121266127299988484109427598488174518496352217959453887222805449754234774047406311190062972867885690337545333241386734824428876595665052730649905438345563755265861552615708879073437575797453583866199963533337412419089655435953207588606278641778796962170113331127330723122473077105157026267033312534656873858029968662447492960990575130231610431505986032191932375986686665729462550074494973895482180866966158871400124560225289463329023158003203107865717692583785722489383582575310485236122345806031170295638219693775725557021395246075440355125182037397932832904906787058458446293665093571394831608228058196856430523943870079872547354630490377779609711239461902213158940243327690712517523949956634958329915667517456561235136264084941567554439241028939457675360665666461435787589523871394662223952554044934516626440096639545618439269571176122084581232485332971012945738452980646590113679635551523851664704244871758716787384234898779560387661249783782828540432505380381386634647402449528848484665911942019101670896495402929284429402780740203023602704687255371136943651372297882768726938003723139881121187286195952230317862099766781825262812603008372419794602922929937898747599900620966720244472569152643576636689820381284142236836196959569023351962642371510071605756169037638070079623744107246519590752269155128567582183956223303512152904913047565142591748719342370033442053753508453610866869872396427794527005625653697420388066918628949402733804694132268212373557711813476959828289826946702076456908260438602358939869273814579996992877963466252645279399558151797086836426507662176650622101091765863038365485729670180796453580683300470520332849095474630492918463590893542982052276468848930296601101148807798164517316001564440748861266913121974321388293545295451773369847571695987548978286874939719896092884993594170191855069466364212505889247892116095830416033601231622767481887088344356841750382688789425686747649964440245827707772041963522514814709937272370531966327995797570767921536896762709263621282760796901763788002110751033796373667296011010090301117749957282543953832418176805255359271417587487729757794050272183267805512738817321338151519351073145906597581150147105290256677168294512857123815871553810395414696290885831639140000017539634748908816554433343737614172324610275274683392083260771550016522608531965843360496808501095006926611557096770426301576687854534977811291878948948408512866952207175828691381005826058089704996618035267070823511406394082342566663127874435994182693228367519969700281286372257203292695958541105666356242435649879854087255780911046270473966908641345197373245619931216360081403715345364307240863018415650977777055012491764833232374649089758909188160042345029275972259069074339266811895845111807797962745773882070712359053234594457346762816540016199275257887862812674528915489273193116046421352415741086621644568565259757043925255675413934473068651554902739873396969327584010700037340127168533830007987247425120861438845415803757465819518124260515185934826322031284057677437216850861663452241522153326157018711538732940716725184988568196564831492924283633263276513226362697899319271458766692079400390992471720867261907023373944555918677484045126341380571544109434846211711363779340033354359166948748886080467755524897551844457106673690160370758577181864979421394685653644686979036146728006305619445349804822159051415184860975932526243528366699949492017584956033845028804291251568545967668643487353983285781728293515659554129862171482132983083118158927218361515984877392780491882244076796924249063554281083400521572192804204423336654988356230863649791993984937218808541487652438509160227279532377507091487281161853957315564309384495805351759826276673863781330209869498414639560605616322592174259773960905711242339462610205342987382612995499472831445261312934827033362929332202230500522419419466689723584029332620597440691109665198073523265531248309561045211740489496741281201245777158059218975787707875252856170589517974291344892386962665601316498266074704018444136491934695315918581382398696220515293438043279672435083897853875802508770856317868860071869945542561447522066117576473715466197351613948896085822488873212889729916925636057407625640027639439977625871673342275714520739387902148206883682137642291242788351636172153269713587215629404738311294261657322783414637344240511488283786128207661564326725003350839555269952609524886532029422165309933333666783383432634352505934359817715569078486137377986381924133644538962141784385366488037234450740061542191803331557364200123155188084049522040831357518712035765213314972012006132326357393965961480653715068071824352510256230422104207485432817019022751907553087438302832049784312462309307900430726005555046933289250805242439264769242386213509333199311730123758798591353831242190106316093583103086890572486640158560516828096784715008029987131082059166670066520955945775634714077887607435444254993090775020519164359604263719726516700033876472242274355629189587560127606928112196943147177232587433972872174097551833117281349285277018402888163282481701813873111290814359718997412915858269468985293331964894251353195352344775676777746315273188163631731694061224296380190502887339265421949782229948442342632747707610426257714288232056973262149353207144094015975073511463473085414023611090631552477664933719990814324258017549263537660205157459432774288039854326693764174038449040441878183282640556785984056709556298147590312358012663694167966318140615072040800472566521307797330993825700450931534198570896305883528962126102232646741363356427706592146539524972674123078061880505149647979669647783000961973849188034364027295786543019825290392125511775945095676362111981558227533626811303158218777770252136867544861144547801245880123585055459889336908274472713328292147242620375159297516001664735496378853987735339770050637852928895715350696620172600305954928756127683697719488520528822548981465501955556580692543980514678740910646476458400407145869533204918406948436080508914943513821652770742678116032287035460610382518538224691140849591057377425507824271582383852803766372944038455478233802172190824846794399277876292846742550120799088566169383156021999098742342805432361728558796361599878920400816068717368767878887784342253056754424670350979659029596568343694523633359997585106001918773616851953950723276979853081265106332598156024469826026074679481628147013369898034120768488751952432966760276336758866693811237319066300116142706282163881826161325158625920638178489959639420924591866818149444124730489342144237640757144427859659035711106302101535035521593963756440029523789617893748632604930608099596441314851275097035783406781828037736828511085139276156579398497859386966836873506638029437020474005376759822222318654969646649170890775811465785888237428529766179161204348395483807983301595448822390859901840639950924463460826420973746060327371792764998492019121604629835863004003459535411598514649296365381986205462761231579484246766044506630243611290336467123753377595970487941475346556874607176890626351390629835329443105880985435642035021888506652340769853281241437979094305258812116097514009849478476478431645734011778585558408142210947668261599182723413780291165621810086833152363317901379957702188388519220959084625869093196405873357988902539297953528473498339052976096866848190554590630310448199200852255578314587469811079587528769390555211280895863459474670623444429951154557605802173773852614453202570794815606686538241026497769134446374802559145776757608520678607366043079999465729544104790474046534844654168181540301220819008381480817638513097542392882143844907763680242655017291885953945546558037071075263671824690099916993344483060236884934283020396100772379608856700328757903539841639070807420502536941712720007193226288927064000287025062034765556938255673500167633473536576117718449196222545728223780117403613907335829523913072435692051719272948281814751532405678270901083230841139274588154004883140969666863785647317156157057281649374433999960472743657580786946031623660895208758484397445248568368626234909299413194078570691962876581280838521891910064781575921752433844152042135027266777659670061256295488804164932412588956852668345726015156632908933425085879542907720258747215579379762406257541569142542134954986264315685914857836257668959967717000926081706318541107443030832121474901612266181141156680201564243345623304100429527781747861809650556262969855182118383398594103068502841628665452576207013149774625999640517060562

[greg7mdp]

geth_log_dl.txt
geth_log_dm.txt

[rjl493456442]

Invalid header encountered               number=17,243,765 hash=c4b4d5..d4945e parent=92b4d0..33efde err="invalid baseFee: have 225385492977903392797341713101650359454085904060009247641194851890689011056244263018550899821681534115504985372092178313730278982394136079439746904696340152838187850488525791796513256252598328946956915155250647913378155959404195503935897141092469973592841118051248790346978995252954243414912460351685096281729075095211291003626922077881627139612360950399392629749335668536862991904857766740022685227187332503305933160912569457284717354130847147977549249805493846577823926833612141710051

This block is invalid, the correct one should be https://etherscan.io/block/17243765

Looks like your ancient store is fired

[greg7mdp]

Invalid header encountered               number=17,243,765 hash=c4b4d5..d4945e parent=92b4d0..33efde err="invalid baseFee: have 225385492977903392797341713101650359454085904060009247641194851890689011056244263018550899821681534115504985372092178313730278982394136079439746904696340152838187850488525791796513256252598328946956915155250647913378155959404195503935897141092469973592841118051248790346978995252954243414912460351685096281729075095211291003626922077881627139612360950399392629749335668536862991904857766740022685227187332503305933160912569457284717354130847147977549249805493846577823926833612141710051

This block is invalid, the correct one should be https://etherscan.io/block/17243765

Looks like your ancient store is fired

I did sudo geth --datadir /var/lib/goethereum removedb twice, answering y when asked whether to remove the ancient chain, and the sync still failed afterwards.

Is doing removedb not enough to remove the ancient db?

[rjl493456442]

Delete ancient chain should be enough, as it basically wipes your entire datadir. You can also try to delete the entire datadir folder manually and resycn from scratch.

[greg7mdp]

Delete ancient chain should be enough

Clearly that wasn't enough, but after I manually deleted the datadir folder geth finally synced successfully.
Usability is not very good.

[greg7mdp]

Well, my node did sync, but after being fine for a couple hours I started having errors again:

Jul 17 00:19:14 gp-nuc-staker geth[460843]: ERROR[07-17|00:19:14.894] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
Jul 17 00:19:14 gp-nuc-staker geth[460843]: INFO [07-17|00:19:14.894] Trie missing, state snapshotting paused  root=38ff0a..99c918 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,857,503      slots=169,043,984      storage=12.94GiB   dangling=0 elapsed=1h23m26.747s eta=24h1m15.489s
Jul 17 00:19:25 gp-nuc-staker geth[460843]: INFO [07-17|00:19:25.664] Resuming state snapshot generation       root=7e17fb..0069eb in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,857,503      slots=169,043,984      storage=12.94GiB   dangling=0 elapsed=1h23m37.517s eta=24h4m21.507s
Jul 17 00:19:25 gp-nuc-staker geth[460843]: INFO [07-17|00:19:25.696] Imported new potential chain segment     number=20,323,813 hash=edc2a7..cbc7ca blocks=1         txs=226         mgas=23.438  elapsed=380.357ms    mgasps=61.621  snapdiffs=338.73KiB triediffs=187.77MiB triedirty=4.42MiB
Jul 17 00:19:25 gp-nuc-staker geth[460843]: INFO [07-17|00:19:25.879] Chain head was updated                   number=20,323,813 hash=edc2a7..cbc7ca root=d96d60..22c9c2 elapsed=6.82122ms
Jul 17 00:19:27 gp-nuc-staker geth[460843]: ERROR[07-17|00:19:27.524] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
Jul 17 00:19:27 gp-nuc-staker geth[460843]: INFO [07-17|00:19:27.524] Trie missing, state snapshotting paused  root=7e17fb..0069eb in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,876,844      slots=169,351,803      storage=12.97GiB   dangling=0 elapsed=1h23m39.376s eta=24h4m53.615s
Jul 17 00:19:37 gp-nuc-staker geth[460843]: INFO [07-17|00:19:37.123] Resuming state snapshot generation       root=8ec6e2..af0c46 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,876,844      slots=169,351,803      storage=12.97GiB   dangling=0 elapsed=1h23m48.975s eta=24h7m39.407s
Jul 17 00:19:37 gp-nuc-staker geth[460843]: INFO [07-17|00:19:37.154] Imported new potential chain segment     number=20,323,814 hash=1034c6..60c9f8 blocks=1         txs=118         mgas=11.603  elapsed=225.066ms    mgasps=51.552  snapdiffs=326.35KiB triediffs=187.72MiB triedirty=5.00MiB
Jul 17 00:19:38 gp-nuc-staker geth[460843]: ERROR[07-17|00:19:38.858] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
Jul 17 00:19:38 gp-nuc-staker geth[460843]: INFO [07-17|00:19:38.858] Trie missing, state snapshotting paused  root=8ec6e2..af0c46 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,896,185      slots=169,659,622      storage=12.99GiB   dangling=0 elapsed=1h23m50.711s eta=24h8m9.391s
Jul 17 00:19:40 gp-nuc-staker geth[460843]: INFO [07-17|00:19:40.663] Chain head was updated                   number=20,323,814 hash=1034c6..60c9f8 root=b0213e..e648a9 elapsed=3.083858ms
Jul 17 00:19:48 gp-nuc-staker geth[460843]: INFO [07-17|00:19:48.866] Resuming state snapshot generation       root=d5112c..69e397 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,896,185      slots=169,659,622      storage=12.99GiB   dangling=0 elapsed=1h24m0.719s  eta=24h11m2.247s
Jul 17 00:19:48 gp-nuc-staker geth[460843]: INFO [07-17|00:19:48.902] Imported new potential chain segment     number=20,323,815 hash=edb71a..d06d65 blocks=1         txs=141         mgas=14.953  elapsed=284.691ms    mgasps=52.523  snapdiffs=318.13KiB triediffs=187.81MiB triedirty=5.77MiB
Jul 17 00:19:49 gp-nuc-staker geth[460843]: INFO [07-17|00:19:49.094] Chain head was updated                   number=20,323,815 hash=edb71a..d06d65 root=70b452..7bfcf0 elapsed=13.27736ms
Jul 17 00:19:50 gp-nuc-staker geth[460843]: ERROR[07-17|00:19:50.624] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
Jul 17 00:19:50 gp-nuc-staker geth[460843]: INFO [07-17|00:19:50.624] Trie missing, state snapshotting paused  root=d5112c..69e397 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,915,526      slots=169,967,441      storage=13.01GiB   dangling=0 elapsed=1h24m2.476s  eta=24h11m32.593s
Jul 17 00:20:01 gp-nuc-staker geth[460843]: INFO [07-17|00:20:01.598] Resuming state snapshot generation       root=f65ee7..6d9ae1 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,915,526      slots=169,967,441      storage=13.01GiB   dangling=0 elapsed=1h24m13.451s eta=24h14m42.151s
Jul 17 00:20:01 gp-nuc-staker geth[460843]: INFO [07-17|00:20:01.629] Imported new potential chain segment     number=20,323,816 hash=e53272..0f9541 blocks=1         txs=137         mgas=16.034  elapsed=323.203ms    mgasps=49.609  snapdiffs=333.13KiB triediffs=188.09MiB triedirty=6.29MiB
Jul 17 00:20:01 gp-nuc-staker geth[460843]: INFO [07-17|00:20:01.776] Chain head was updated                   number=20,323,816 hash=e53272..0f9541 root=b0334e..193bfb elapsed=8.542581ms
Jul 17 00:20:02 gp-nuc-staker geth[460843]: WARN [07-17|00:20:02.852] Blob transaction swapped out by signer   from=0x7f9D9c1BCE1062E1077845eA39a0303429600a06 nonce=509   id=268,435,460
Jul 17 00:20:03 gp-nuc-staker geth[460843]: ERROR[07-17|00:20:03.304] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
Jul 17 00:20:03 gp-nuc-staker geth[460843]: INFO [07-17|00:20:03.304] Trie missing, state snapshotting paused  root=f65ee7..6d9ae1 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,934,867      slots=170,275,260      storage=13.03GiB   dangling=0 elapsed=1h24m15.156s eta=24h15m11.599s
Jul 17 00:20:13 gp-nuc-staker geth[460843]: INFO [07-17|00:20:13.088] Resuming state snapshot generation       root=8bc370..ab8ff8 in=0e02bb..03300f at=f92c9b..a53be2 accounts=19,934,867      slots=170,275,260      storage=13.03GiB   dangling=0 elapsed=1h24m24.941s eta=24h18m0.604s
Jul 17 00:20:13 gp-nuc-staker geth[460843]: INFO [07-17|00:20:13.120] Imported new potential chain segment     number=20,323,817 hash=10c91a..10ba1f blocks=1         txs=179         mgas=19.202  elapsed=286.562ms    mgasps=67.008  snapdiffs=334.69KiB triediffs=188.16MiB triedirty=7.09MiB
Jul 17 00:20:13 gp-nuc-staker geth[460843]: INFO [07-17|00:20:13.315] Chain head was updated                   number=20,323,817 hash=10c91a..10ba1f root=a60474..aada29 elapsed=22.789035ms
Jul 17 00:20:14 gp-nuc-staker geth[460843]: ERROR[07-17|00:20:14.754] Unexpected trie node                     location=disk owner=0e07ce..82eb44 path=[]                                 expect=1ff3d4..c3b173 got=c5d246..85a470 blob=nil
J

My NVMe drive seems fine.

[rjl493456442]

Do you have the logs since the initial state sync started?

Your state data is fired somehow, unfortunately you have to remove the state data and resync

[rjl493456442]

Could be the same thing as #29436

[greg7mdp]

Do you have the logs since the initial state sync started

yes, I do. It is a pretty large file.
Is there a way to find where the last sync started? The logs have multiple sync attempts.

[rjl493456442]

Please share the log as it could be useful for debugging. Thanks and Best regards Gary rong Gregory Popovitch ***@***.***>于2024年7月19日 周五上午11:45写道：

…

Do you have the logs since the initial state sync started yes, I do. It is a pretty large file. — Reply to this email directly, view it on GitHub <#30149 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNO6OPNT4BAKGHL4XZNAT3ZNCDUBAVCNFSM6AAAAABK2IZSDWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZYGA2TQOJQGI> . You are receiving this because you commented.Message ID: ***@***.***>

[greg7mdp]

Please share the log as it could be useful for debugging.

What's the best way to share the log? It is a multiple GB file.

[rjl493456442]

Ah? Are you sure the geth log file is multiple GB? It's not the datadir, just the runtime log

Btw, do you still have the node data available by any chance? If so, please run the command and reply the result here

db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4400
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4401
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4402
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4403
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4404
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4405
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4406
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4407
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4408
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb4409
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440a
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440b
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440c
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440d
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440e
db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440f

[greg7mdp]

Btw, do you still have the node data available by any chance? If so, please run the command and reply the result here

Sure, here it is:
geth_data.txt

[rjl493456442]

Could you double check this command?

geth db get --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb440a?

Btw, you could try

geth db put --datadir <your-dir> 0x4f0e07ce814e92c6d4b4a449b9106c4622276a6b7e68101f29f4dab23d0882eb44 0xf90211a02a94c51054c076eef40db24c617065b8c4946c2fbcd79a55045c3865423c69a3a0ed1e3f6d9091d7242e32844ca4c18117d23b1fa199e190cef4081381e614b25ea05c0b7e9c42bbde42ce576726b664c9e414dd9a3b4a34f72c0f4b6bc623bb30f5a0bd51a1ef1ede2265d8692c0ccff84747567342bd47f39d667e3d622af987f1fda064b8634953677b0ec62775ffc2fdc87c41875edff41aacf34e53393aae6ad698a0799776316ddd722503508e4f6a50c792810e3b121bbe9ff16ccc6467af246deea016f566f7d3e518e6a272c37fab66e2cbfff42b4f5716102f50d667a1e597474ca09912e0ad6dfe2e649f61c90af78f654e0fcee5a0c068111210aa1ef25fa72238a0110597c3b4ad47f7990278cddbc72a55853b23924f953018d890af2f702c3d67a0b319c4244a59cd374faf0fa526b324c4c84d0b14f53dbe888d023eaf4cabf0c3a0b175dfb4d5054bc152612fa04bd0650d7413dde20f25246c65c8d8fca44c53e6a09a23d718a63a53566e87fdcdc91c513ef8fccd37c005db6bdff42a8a7ff277faa0c8bc9ca45d8f948b2a91e43b50cb1ff0811170a7d3a445a4dcfa3e59b9cf69fba0ebdc864b666a504a8cc9fefc97f4a2b7dbab8fe5f64e69104992ed4770c7c105a06ddfcc0d47d253383be0dc586f123283a128a23dfd6cf01a2f1f196194515db0a0739634d64a37aa365ef73b7f1ab251d555f6ad8a2ef247d51d9179170c8e74c980

It will inject the missing entry and let's see if your node can recover from it