Audit log: background job to clean up entries that never completed

[david-crespo]

Followup to #7339.

Unlike with audit log entry initialization (because we bail if it fails), we do not have a guarantee that audit log completion runs successfully because we don't want to turn every loggable operation into a saga to enable rollbacks. While failures should be unlikely (and we do have retries in place), they are still possible, and we really don't want there to be anything missing from the audit log.

To deal with this, we need a background job to complete any rows hanging around uncompleted for longer than N minutes or hours. Because these will not have success or error info about the logged operation, they will use the Timeout variant, which was already added for this purpose in #7339.

omicron/schema/crdb/dbinit.sql

Lines 5776 to 5783 in 6ab7e96

	CREATE TYPE IF NOT EXISTS omicron.public.audit_log_result_kind AS ENUM (
	'success',
	'error',
	-- represents the case where we had to clean up a row and artificially
	-- complete it in order to get it into the log (because entries don't show
	-- up in the log until they're completed)
	'timeout'
	);

omicron/nexus/db-model/src/audit_log.rs

Lines 78 to 81 in 6ab7e96

	// Enum values
	Success => b"success"
	Error => b"error"
	Timeout => b"timeout"

omicron/nexus/types/src/external_api/views.rs

Lines 1603 to 1612 in 6ab7e96

	// Note that the DB model result kind analogous to Unknown is called Timeout
	// -- The name "Timeout" feels useful to write down for the DB but also
	// feels like too much of an implementation detail to expose to the user --
	// it makes it sounds like the operation timed out rather than the audit log
	// entry itself.
	/// After the logged operation completed, our attempt to write the result
	/// to the audit log failed, so it was automatically marked completed later
	/// by a background job. This does not imply that the operation itself timed
	/// out or failed, only our attempts to log its result.
	Unknown,