keep the system from running out of disk space

[davepacheco]

(This high-level issue might well get turned into smaller issues and then closed.)

Recent customer issues have highlighted several problems around storage accounting and space management.

Problems

• The process of allocating Crucible regions assumes that Crucible can use the whole disk. This does not account for other uses of U.2 devices: namely, the root filesystems and persistent datasets of control plane zones and Propolis zones. (Anything else?)
• The process of allocating Crucible regions does not account for Crucible overhead (a fudge factor of about 1.2-1.25x).
• We don't have quotas or reservations in most places, so if something does go haywire, it can take a lot out with it.
• There appears to be no process that clears old data out of debug datasets. We have deployed systems where some disks have upwards of 1 TiB of space used by these files.
• While investigating this, we discovered crypt/debug dataset not mounted #7874.

All of these can cause systems to run out of disk space, triggering various ugly failures.

Proposed steps

For R14

• make a quick, conservative guess for how much storage space we can safely give to Crucible while leaving enough for other consumers - outcome summarized here.
• update Crucible region allocation to use this as the total size available to Crucible rather than the pool size
  Prevent region allocation from filling pools #7912
• update accounting data for Crucible regions to reflect the 1.2-1.25x fudge factor (specific proposal was: create a new column with this information so that the table reflects both actual space requested and what we've allocated for it) Account for Crucible Agent reservation overhead #7885
• update the Crucible region allocation to use the value with-fudge-factor instead of the requested size of each region Account for Crucible Agent reservation overhead #7885
• fix debug dataset mount issue [sled-agent] Ensure that datasets get mounted #7887, [sled-agent] Make new mountpoints immutable #7888
• add omdb tool to identify pools that are overprovisioned, when considering the Crucible fudge factor and the max space allowed for Crucible Prevent region allocation from filling pools #7912
• create support runbook that uses omdb to identify regions that should be replaced and trigger replacement for them (and plan to do this during R14 upgrade) --- PENDING (may punt this if the number of overprovisioned pools is small; the tools will be necessary when we set quotas on crucible datasets)
• create support runbook for clear out large debug directories and reserve clickhouse dataset + 2 spares (and plan to do this during R14 upgrade) https://github.com/oxidecomputer/customer-support/issues/333

Longer term

• for each consumer of disk space, on all disks (this would include: crash dumps, core dumps, GZ data, zone root filesystems, zone persistent data, etc.)
  • make an estimate of how much disk space it needs
  • figure out what steps might be necessary to keep that under its limit (e.g., log rotation, archiving, deletion)
  • update the estimate we made above (for how much Crucible is allowed to use) to reflect this
• for crucible regions, implement measures to prevent unbound snapshot usage (e.g., pantry scrubber) so we can shrink region dataset's 3x quota
• update Reconfigurator to apply reservations and/or quotas to implement those limits
• update Reconfigurator to look at the storage allocated and required when placing components (so that we don't overprovision storage on disks)
• figure out why debug datasets are sometimes not mounted and how to fix it (see: crypt/debug dataset not mounted #7874)
• implement some kind of monitoring for disk space usage (raise an active problem if datasets get full)
• review/improve policy for deleting files from the debug dataset (current logic)

[askfongjojo]

• There appears to be no process that clears old data out of debug datasets. We have deployed systems where some disks have upwards of 1 TiB of space used by these files.

So, this process actually exists. It was spec'd out in #2478 and implemented in #3735 (slightly after the first customer ship). The housekeeping is clearly not happening for some reason. The PR includes some dataset mounting changes and I don't know if that might explain #7874.

[askfongjojo]

Related backlog items and discussions:

• Tracking Issue for Managing Low Disk Space Conditions #1700 (RFD 312)
• Tracking Issue for "zones can run out of resources" #7468
• Nexus: Account for all storage on U.2s #6110
• Nexus overprovisions disks with control plane zones on them #7225 (comment) (pt 2-4)
• Consider adding a quota to the crash dataset on the M.2 drives #7882

[askfongjojo]

Some data from racks in the field (disk usage per zone, from r13-reconfigurator-state.json):

customer 1

• clickhouse: 270G
• cockroachdb: 5G

customer 2

• clickhouse: 190G
• cockroachdb: 5G

customer 3

• clickhouse: 200G
• cockroachdb: 5.5G

customer 4

• clickhouse: 200G
• cockroachdb: 5G

customer 5

• clickhouse: 140G
• cockroachdb: 5.5G

dogfood

• clickhouse: 145G
• cockroachdb: 11G

colo

• clickhouse: 305G
• cockroachdb: 11G

other zone types:

• nexus: 0.6G
• oximeter: 0.4G
• internal_dns: 0.3G
• external_dns: 0.3G
• pantry: 0.3G
• ntp: 0.3G
• propolis: 0.32G

[jmpesp]

I've opened #7885, which addresses points 3 and 4 of the "Before R14" list.

[askfongjojo]

Re point 1, based on the non-trivial usage data we saw in the field , the options discussed are:

1. Set aside 500G per disk for control plane use: This makes it easier to replace clickhouse if we have to expunge a bad sled/disk that runs it. But it is unnecessarily conservative and takes away a substantial amount of crucible capacity.
2. Set aside 150G *250G per disk for control plane use, and mark the crucible dataset colocated with clickhouse non-provisionable. This requires implementing Want ability to take a crucible dataset out of provisioning pool #3480 in r14 (invoked from omdb, not necessarily an operator API yet).
3. (a variation of option 2) Have two more disks marked non-provisionable as spares for clickhouse.
4. Set different limits for different disks/deployments, depending on which omicron zones run on the disks and their sizes. This is deemed too hard to manage and can be error prone.

Option 3 seems the most reasonable one at this time. (*4/3/2025: Option 2 is revised to 250G based on James' comment below)

Re point 2, the long-term implementation is to have the crucible quota as dataset config. The short-term solution can have the size available hardcoded or stored in the database temporarily. Hardcoding is mostly ruled out because changing it in the field would require a software update. The size avail db setting will perhaps be named in a way to signify that no new code should rely on it being there permanently.

[jmpesp]

In light of #7902, 150G may be too low: if the difference between the zpool and zfs outputs is about 100G, we should maybe bump that to 250G accordingly.