[Execution Node] Reduce Temporary Linux Page Cache Spikes by 150+ GB RAM

[fxamacker]

Problem

As noted in PR #2336, we can further reduce Linux Page Cache use by evicting portions of the file during use rather than waiting to have Linux evict the entire file.

To keep things simple, PR #2336 made Linux evict entire file from page cache after using large (98GB), which successfully prevents Linux eventually hoarding hundreds of GB in its page cache. We can improve upon it by reducing the temporary spikes in Linux Page Cache.

Updates epic #1744

The Proposed Solution

As mentioned in Other Approaches section of PR #2280, it is possible to remove segments of the file while it is used so we can reduce peak Linux page cache usage by 98GB. This can be done without using O_DIRECT so we can avoid edge cases like alignment issues or incompatibility with some filesystems.

Optionally, remove additional files from the Linux page cache such as WAL segment files.

[fxamacker]

I'm closing this because:

• PR [EN Performance] Reuse ledger state for about -200GB peak RAM, -160GB disk i/o, and about -32 minutes duration #2792 eliminates the need to read checkpoint file (except during EN startup)
• there's a chance we'll split the checkpoint file into smaller files so it won't be necessary to evict segments of the file from Linux page cache during use