EthMainnet: download stuck

[AskAlexSharov]

anacrolix/main-torrent-perf

DOWNLOADER_MAX_CONNS_PER_HOST=20 go run ./cmd/erigon --datadir /erigon-data/remove_me_experiment --chain mainnet --prune.mode=archive  --persist.receipts --nat=stun --pprof --pprof.port=6061 --metrics --metrics.port=6162 --http=false --txpool.disable --log.console.verbosity=4 --log.dir.verbosity=4 --torrent.upload.rate=0 --torrent.webseed.download.rate=1G --torrent.download.rate=0 --torrent.port=0 --torrent.verbosity=3 > /erigon-logs/erigon-process2.log 2>&1 &

[INFO] [07-31|03:37:39.818] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:37:41.813] [snapshots] downloading                  len=269 hashed=3.5GB hash-rate=331.6MB/s completed=6.8GB completion-rate=443.9MB/s flushed=0B flush-rate=0B/s downloaded=6.8GB download-rate=443.8MB/s webseed-trips=311 webseed-active=6 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=8.0GB
[INFO] [07-31|03:37:41.814] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=13/269 data=6.8GB time-left=11s total-time=18s download-rate=443.8MB/s hashing-rate=331.6MB/s alloc=1.4GB sys=1.5GB
[DBUG] [07-31|03:37:41.886] [downloader] Collecting...               from=23036454 fromHash=0xb30aacc22c4e21adf412931cdfed3f95db0884fb4ce21fa7ced812ef1c4035c5 to=23036454 toHash=0xb30aacc22c4e21adf412931cdfed3f95db0884fb4ce21fa7ced812ef1c4035c5 len=1 fromPeer=3a65d441a0947194c1095dec2b6761c4a430470f4d3111bd56cbc378bccb8ca37b65dd16f261d522fe9f600bdb5b7addb5fb856af6f1d0f103f45d85e833c018
[DBUG] [07-31|03:37:41.886] [downloader] posAnchor is nil
[INFO] [07-31|03:37:55.032] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:37:55.595] [snapshots] downloading                  len=269 hashed=7.3GB hash-rate=263.6MB/s completed=11.4GB completion-rate=315.6MB/s flushed=0B flush-rate=0B/s downloaded=11.4GB download-rate=315.7MB/s webseed-trips=396 webseed-active=7 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.4GB
[INFO] [07-31|03:37:55.596] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=32/269 data=11.4GB time-left=0s total-time=32s download-rate=315.7MB/s hashing-rate=263.6MB/s alloc=2.0GB sys=2.0GB
2025/07/31 03:38:22 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
[DBUG] [07-31|03:38:23.447] [downloader] Collecting...               from=23036457 fromHash=0xa8b3722ca8bfafecf3020f9e41c498c50c3e928193ddc9c04e9ee3bad87ca844 to=23036457 toHash=0xa8b3722ca8bfafecf3020f9e41c498c50c3e928193ddc9c04e9ee3bad87ca844 len=1 fromPeer=af99cee6d6084514bc1ae4859a6f2d61f7f51efb24d4f8b9acea009597614f69f291c3193b4e850e40005cc420376d05a2a790f8b7af7a85f97c6c487eab1372
[DBUG] [07-31|03:38:23.447] [downloader] posAnchor is nil
[INFO] [07-31|03:38:24.367] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:38:24.368] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=150.4MB/s completed=11.6GB completion-rate=8.4MB/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=8.4MB/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:38:24.368] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=0s total-time=1m1s download-rate=8.4MB/s hashing-rate=150.4MB/s alloc=2.1GB sys=2.5GB
[DBUG] [07-31|03:38:31.640] [downloader] Collecting...               from=23036458 fromHash=0x3f0de54a41a74b4458519f8ca3a51f5625db3011b9a770dd7ca7d2de9818629c to=23036458 toHash=0x3f0de54a41a74b4458519f8ca3a51f5625db3011b9a770dd7ca7d2de9818629c len=1 fromPeer=64fc2b2e58d35cb05f436c214f3f05195f450219b15c9344204aaad04ec4dea0d7d7beffae449f0072bd2658079a792b57038d55f0487413c7eb01965f3b9580
[DBUG] [07-31|03:38:31.640] [downloader] posAnchor is nil
[DBUG] [07-31|03:38:32.978] [downloader] Collecting...               from=23036458 fromHash=0x3f0de54a41a74b4458519f8ca3a51f5625db3011b9a770dd7ca7d2de9818629c to=23036458 toHash=0x3f0de54a41a74b4458519f8ca3a51f5625db3011b9a770dd7ca7d2de9818629c len=1 fromPeer=fd378aa661d10fd45a7f40cc7ad0eca349a4f1266ea85caee0270a8a55f56c7deddf2c388a4721406fcd17bb9a8cc3319165aeecf5c5befeb8b5dfc83d151ba3
[DBUG] [07-31|03:38:32.978] [downloader] posAnchor is nil
[DBUG] [07-31|03:38:38.439] [downloader] Collecting...               from=19361417 fromHash=0x865127890391f98e7ff2a3320e3f7350ab321bf38bf80ecc0e01ce83501c638d to=19361417 toHash=0x865127890391f98e7ff2a3320e3f7350ab321bf38bf80ecc0e01ce83501c638d len=1 fromPeer=c2a76f2e6c9842a89676fb82f803ef8c2c4dbc4063b5f2614ff1e2e5b56808813a25e5742f3bdef45b6d90b62cf30e9a05b59e366ea9337adc48560e947a173e
[DBUG] [07-31|03:38:38.439] [downloader] posAnchor is nil
[DBUG] [07-31|03:38:52.161] [downloader] Collecting...               from=19426589 fromHash=0xcb5185a8565486c2fd3b91551d2f728712fde021186bd67161f91e4ca2912fbd to=19426589 toHash=0xcb5185a8565486c2fd3b91551d2f728712fde021186bd67161f91e4ca2912fbd len=1 fromPeer=a89544be02d86d73bca3a56af3f894b3c3a68796f76c690fb68bcc1f2376c8be6bf6e55c69ebf8ff4a9e81f13bd71364b58de6a166d709ae012682e85cc7df47
[DBUG] [07-31|03:38:52.161] [downloader] posAnchor is nil
[INFO] [07-31|03:38:54.365] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:38:54.365] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:38:54.365] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=1m31s download-rate=0B/s hashing-rate=0B/s alloc=2.6GB sys=3.3GB
[DBUG] [07-31|03:39:16.885] [downloader] Collecting...               from=23036462 fromHash=0xb1b4095d934b984d6912793ab0d53f3ec0dc209128490f53c0bcde861455736f to=23036462 toHash=0xb1b4095d934b984d6912793ab0d53f3ec0dc209128490f53c0bcde861455736f len=1 fromPeer=9478c6bb79f6121ce9af5d85600c7a144207df54df86869dc8cf93cdc643531f99dbcd655505e4f06d00c5d5ac97223f166df00e82197eca10eaf76dc2ef0332
[DBUG] [07-31|03:39:16.885] [downloader] posAnchor is nil
[DBUG] [07-31|03:39:21.021] [downloader] Collecting...               from=19426600 fromHash=0x88ee69929ab8a4c5d61c92e26dfad41f8dbbed1bcf15b14750ad3e9588f0c625 to=19426600 toHash=0x88ee69929ab8a4c5d61c92e26dfad41f8dbbed1bcf15b14750ad3e9588f0c625 len=1 fromPeer=31ce05ce5b3e7b502fb133006c77720e44f3c61027c4111eb3d796e63f719a1d91d104ef2e7c077520c0fd4619c9e6378f5965e01b12b17f1ab7b9d462817358
[DBUG] [07-31|03:39:21.021] [downloader] posAnchor is nil
[DBUG] [07-31|03:39:23.857] [downloader] Collecting...               from=22345027 fromHash=0xe57e9da923d56b6392eed553c788966a650b23af2748de42ab19a72840b298b8 to=22345027 toHash=0xe57e9da923d56b6392eed553c788966a650b23af2748de42ab19a72840b298b8 len=1 fromPeer=cb17ba63bede19a93b63480ba7939a3a5aca17486bfed59206c72601972238539e8c88343a0b4ed3dc51eaa141b1344fe18a0b6a44cde9ea0c6dda81ea67915c
[DBUG] [07-31|03:39:23.857] [downloader] posAnchor is nil
[INFO] [07-31|03:39:24.366] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:39:24.367] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:39:24.367] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=2m1s download-rate=0B/s hashing-rate=0B/s alloc=1.8GB sys=3.4GB
[INFO] [07-31|03:39:54.366] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:39:54.366] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:39:54.366] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=2m31s download-rate=0B/s hashing-rate=0B/s alloc=2.7GB sys=3.6GB
[INFO] [07-31|03:40:23.340] [mem] memory stats                       Rss=3.5GB Size=0B Pss=3.5GB SharedClean=4.0MB SharedDirty=0B PrivateClean=111.1MB PrivateDirty=3.4GB Referenced=3.5GB Anonymous=3.4GB Swap=0B alloc=2.1GB sys=3.7GB
[INFO] [07-31|03:40:24.365] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:40:24.365] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:40:24.366] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=3m1s download-rate=0B/s hashing-rate=0B/s alloc=2.1GB sys=3.7GB
[INFO] [07-31|03:40:54.367] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:40:54.367] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:40:54.368] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=3m31s download-rate=0B/s hashing-rate=0B/s alloc=2.9GB sys=3.7GB
[INFO] [07-31|03:41:24.368] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:41:24.368] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:41:24.368] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=4m1s download-rate=0B/s hashing-rate=0B/s alloc=1.9GB sys=3.7GB
[DBUG] [07-31|03:41:46.820] [downloader] Collecting...               from=23036432 fromHash=0x1aef393b5f0019763c6689b814d4d0e9b205562f6b97cf87a561e4776a3a8bad to=23036432 toHash=0x1aef393b5f0019763c6689b814d4d0e9b205562f6b97cf87a561e4776a3a8bad len=1 fromPeer=2a88a445e00d22f8c3243acda8076079aaaf30232f6333d544c62d0c38b23d478e4ce95f1a213145c72764ba6840cfac08d2b06665d503af17c0a16a6e6dcee6
[DBUG] [07-31|03:41:46.820] [downloader] posAnchor is nil
[INFO] [07-31|03:41:54.366] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:41:54.366] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:41:54.367] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=4m31s download-rate=0B/s hashing-rate=0B/s alloc=1.8GB sys=3.7GB
[DBUG] [07-31|03:42:18.663] [downloader] Collecting...               from=23036477 fromHash=0xdd83930283b4ab3323aca35ce07694e1d4defec957caf474a75e48136e286a60 to=23036477 toHash=0xdd83930283b4ab3323aca35ce07694e1d4defec957caf474a75e48136e286a60 len=1 fromPeer=03390daadfe547ef2f6a0e43d99a9d9c739e635e6753dce0fad1e7c23bbed3d87f63745dd5bc0d243f5f1399578120850831ad668e29246075fedc3b52d99f4d
[DBUG] [07-31|03:42:18.663] [downloader] posAnchor is nil
[DBUG] [07-31|03:42:23.316] [kv] table sizes                         all=ReclaimableSpace:64.0MB,Config:704.0KB,Freelist:16.0KB,BlockBody:16.0KB,Sequence:16.0KB,DbInfo:16.0KB,LastHeader:16.0KB,HeaderNumber:16.0KB,SyncStage:16.0KB,LastBlock:16.0KB,MaxTxNum:16.0KB,HeadersTotalDifficulty:16.0KB,Migration:16.0KB,CanonicalHeader:16.0KB,Header:16.0KB,
[DBUG] [07-31|03:42:23.808] [downloader] Collecting...               from=23036477 fromHash=0xdd83930283b4ab3323aca35ce07694e1d4defec957caf474a75e48136e286a60 to=23036477 toHash=0xdd83930283b4ab3323aca35ce07694e1d4defec957caf474a75e48136e286a60 len=1 fromPeer=6c87c112a66d385e705a3c3845d9a2860b40b3afa3c0efb11d4481cf9fc8524f8774de22b6428883690cb44f3a670f7aa1fca2a62476461111b36f5d538da6af
[DBUG] [07-31|03:42:23.808] [downloader] posAnchor is nil
[INFO] [07-31|03:42:24.367] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:42:24.367] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:42:24.367] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=5m1s download-rate=0B/s hashing-rate=0B/s alloc=2.0GB sys=3.7GB
[INFO] [07-31|03:42:54.366] [snapshots] no metadata yet              files=1 list=v1.0-004000-004500-headers.idx
[DBUG] [07-31|03:42:54.366] [snapshots] downloading                  len=269 hashed=11.6GB hash-rate=0B/s completed=11.6GB completion-rate=0B/s flushed=0B flush-rate=0B/s downloaded=11.6GB download-rate=0B/s webseed-trips=537 webseed-active=1 webseed-max-active=269 webseed-discards=0 webseed-fails=0 webseed-bytes=11.6GB
[INFO] [07-31|03:42:54.367] [1/6 OtterSync] Syncing                  file-metadata=268/269 files=268/269 data=11.6GB time-left=∞ total-time=5m31s download-rate=0B/s hashing-rate=0B/s alloc=2.3GB sys=3.7GB

[AskAlexSharov]

linter is green:

go run ./cmd/downloader manifest-verify --chain mainnet     --webseed 'https://erigon3-v1-snapshots-mainnet.erigon.network'
INFO[07-31|03:51:24.830] console logging only
INFO[07-31|03:51:24.830] Build info                               git_branch= git_tag= git_commit=
INFO[07-31|03:51:24.830] console logging only
## REPORT [OK] on https://erigon3-v1-snapshots-mainnet.erigon.network
 - manifest exist: true
 - missing torrents (files without torrents): 0
 - dangling (data file not found) torrents: 0
SEED [OK] https://erigon3-v1-snapshots-mainnet.erigon.network

-----------------------REPORTS OVERVIEW--------------------------
## REPORT [OK] on https://erigon3-v1-snapshots-mainnet.erigon.network
 - manifest exist: true
 - missing torrents (files without torrents): 0
 - dangling (data file not found) torrents: 0

[bravoadam]

I am having this happen on Polygon mainnet too. It pauses for like 30-60 min and then continues. I used to be able to download all 5.5TB in about 2 hours. Going on 8 hours now and only 50% done.

[anacrolix]

I have the fix for this in my branch. Will push tomorrow. Per my notes it's an issue with the default http2 configuration bundled with Go

[VBulikov]

VBulikov added Imp1 as Importance

[AskAlexSharov]

@anacrolix let’s merge to 3.1 and main?

[anacrolix]

See #16328.

[anacrolix]

This should be fixed by f31dd0f.