Full sync with gcmode=archive extremely io heavy & slow compared to regular full sync, seems to not be using cache

[c2h5oh]

System information

Geth version: 1.8.22-stable
OS & Version: Arch Linux, 4.20.6 kernel
Ryzen 7 2700, 16GB RAM , 2TB SATA SSD

Expected behaviour

gcmode=archive results in more storage being used and slightly to moderately slower than "regular" full sync.

Actual behaviour

with --gcmode=archive:

• cache seems to be unused: logs are showing cache=0.00B. tested with multiple cache sizes ranging from 2 to 14 GB
• A LOT slower. after 40 hours it imported 2.8M blocks. By comparison without archive it imported 2.8M blocks in about 85 minutes
• those 2.8M blocks resulted in a chain that was just over 80GB, despite that about 200GB was read and written PER HOUR
• importing speed dropped to <10 txn per second before reaching block 3M, by comparison without archive I was seeing 500-2000 txn/s. At <10 txn/s it's not going to ever catch up.

Steps to reproduce the behaviour

geth --syncmode=full --gcmode=archive --cache 2048

[holiman]

Sounds like a flaw, even though the caches aren't used for in-memory gc, the read/write caches ought to still be used

[karalabe]

The write cache (which is the primary one used) is flushed when a trie is committed to disk. We could add a modification to move anything flushed to the read cache, but it's a non obvious performance implication.

[holiman]

This one is basically the same as #16237

[holiman]

We (@karalabe) will make a modification to move the memory allowance for the write-cache over to the read-cache, to better make use of available memory.

[karalabe]

Btw, the cache field reported in the logs is the dirty cache (originally when I added the log there was only the dirty cache, no clean cache). I changed the name from cache to dirty in my PR above.

Regarding the huge disk IO, that's leveldb compaction. Trie nodes are hash->value mapping, so every trie node we insert into the database gets dumped at a random location. Since leveldb stores everything sorted by key, the more random stuff we insert, the more maintenance overhead it is to keep them sorted. Hence where the insane disk IO comes from. Unfortunately this is a limitation of the data model Ethereum was designed to use (Merkle Patricia trees). @AlexeyAkhunov is experimenting with storing things in a different format that makes archive nodes less heavy, at the cost of losing provability of past data (i.e. data is available, just not provable in real time).

[holiman]

On our monitoring node, in 48h, current master got through 4.69M blocks -- the first 2.8M blocks took about 8 hours to complete. So on that particular (quite beefy) machine, it was not totally degraded.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[karalabe]

This issue is not relevant anymore due to:

• Archive node caches have been reblanaced way back to use the dirty cache allowance as a read cache too.
• We've fixed leveldb to remove a quadratic io blowup (that actually made archive sync in 1.9 about 6 times faster).