Automatically purge target directories after reaching max size

[pietroalbini]

At the moment the target directories used by Crater don't have a size limit, so they reach hundreds of gigabytes in size, forcing us to have 4TB disks on the agents. We should implement a way to keep them within a configurable size.

@aidanhs suggested removing them after the configured size is reached. This will slow down the runs a lot, but if we keep the max size high enough we can maybe clear them max 1 or 2 times during a run, which shouldn't hit the speed too much.

[Eh2406]

suggested removing them after the configured size is reached.

Is there a way of determining the size of a folder that is faster than iterating over all the files and summing the size?

If so then we can run Cargo-Sweep each time we hit the configured size, it keeps only artifacts newer then a timestamp.

If not I can add a argument to Cargo-Sweep to remove older artifacts until the folder is under a target size.

Also do you happen to know if the target folder support "last access time"? If like my computer it does not then the next step is on Cargo, if it does support it then I can make progress with Cargo-Sweep.

[pietroalbini]

Is there a way of determining the size of a folder that is faster than iterating over all the files and summing the size?

The easiest way is to do whatever cleanup routine we choose after we reach, let's say, 90% of total disk usage on the partition. Querying the free space on a partition should be instantaneous I think.

If not I can add a argument to Cargo-Sweep to remove older artifacts until the folder is under a target size.

This would be great!

Also do you happen to know if the target folder support "last access time"? If like my computer it does not then the next step is on Cargo, if it does support it then I can make progress with Cargo-Sweep.

It's really unreliable on the current machines, and AFAIK making it reliable will slow things down a lot.

[Eh2406]

The lru cleaning has a PR.

The PR to have Cargo maintain the mtime was merged, but was then put behind a feature flag as it broke the playground. @ehuss reports that the feature as implemented slows down the playground by ~11sec, and a more limited version (just enough for cargo-sweep to work) slows it down by 2sec. I think Crater is also using Docker and AWS in a similar way.

[pietroalbini]

The PR to have Cargo maintain the mtime was merged, but was then put behind a feature flag as it broke the playground. ehuss reports that the feature as implemented slows down the playground by ~11sec, and a more limited version (just enough for cargo-sweep to work) slows it down by 2sec. I think Crater is also using Docker and AWS in a similar way.

Yep, we're basically using the exact same setup.
Thanks for all your effort on this by the way!

[aidanhs]

When I've suggested this previously it has been in combination with a proposal that we do some form of sorting of crates with their dependencies to reduce the impact of auto-cleaning (e.g. chances are you'll have finished a bunch of crates that then don't need rebuilding).

More interestingly, one could imagine selectively cleaning out dependencies when they're done (e.g. once you're done with serde version 1.0.9 you clean just that).

[Eh2406]

On discord @aidanhs "I can say that crater doesn't put the target directory in the image so there's no CoW stuff. in fact I think the crater image is entirely read-only and used for linking libraries"

Which suggests that a quick experiment to see how big the overhead is with the existing flag may be worth it, if someone has the time.

I like the idea of a Crater graph aware solution, I think it can build on #193. I can volunteer to help in as much as it involves adding to the "target folder gc" ecosystem. (I am mostly interested in helping that, Crater is just an example at the moment.)

One thought I had is that if Crater is walking over a topological sort of the things to build and knows it is done with all the things it built before "foo" then it can use cargo-sweep with the exact time it built "foo". Or even just dell all files with a creation time before "foo".

I think cargo clean -p "serde v1.0.9" may be all the "gc" needed for your "more interesting" suggestion.

[Eh2406]

discord: @pietroalbini at 12:21 PM

redeployed crater -- the target directory is now automatically being cleared
rm -rf target when we reach 90% disk usage
but that's the quickest solution I could get up and running fast, since we're going over 4tb now

#411