Clearing/Expiration of old values in histogram/limit to histogram

[ported-pw]

This is possibly more of a question and relates to metrics-util, specifically the AtomicBucket struct and it's use in metrics-exporter-prometheus.
Is there currently any intended way of being able to limit values recorded in a bucket (histogram) either by amount of time in either the type itself or the recorder? If I'm not mistaken or fundamentally misunderstanding something, currently using a Histogram would lead to a slow (or fast, depending on the rate of metrics inserted) memory leak, leading to eventual OOM of the application. Even if I clear the histogram with every snapshot, I make the memory usage dependant on an external service polling it.
What would be the best way of going about this to enable using histograms for high rates of insertions, possibly without creating a custom recorder including types to do so?

[tobz]

👋🏻

At a high level, your assessment is accurate: we buffer all individual histogram measurements until something drains them. This indeed represents an unbounded queue, effectively. We used to have a rolling windowed histogram setup where users had to explicitly specify how long a histogram should record for, but that introduced a lot of overhead to recording each point since we had to make every record call check if things needed to be dropped, etc.

Ultimately, the optimal solution likely revolves around using some sort of sketch storage internally to record histogram measurements so that recording is bounded in space. This involves a similarly complex set-up to handle in a lock-free fashion, and isn't something I could get to in the very near future.

Practically speaking, there's three easy workarounds that should help bound the usage, two of which you can do on your own and the third which I would accept a PR for:

• grab the PrometheusHandle before installing the recorder and poll it externally on an interval
• use the push-gateway feature flag to push on an interval instead of scraping
• update the builder to generate a future which both drives the Hyper server future as well as a call to get_recent_metrics on an interval

[drmingdrmer]

but that introduced a lot of overhead to recording each point since we had to make every record call check if things needed to be dropped, etc.

Is it possible to add a counter and only check the need for a drop for every 1024 counts? Therefore, the amortized overhead should be very low.

[tobz]

It's certainly possible for exporters to choose a scheme where they bound the number of samples they'll hold, or use a windowing mechanism to only keep the samples from the last N seconds, or to drive consumption of AtomicBucket on an interval, and so on.

These are things I'm willing to review/potentially accept PRs for.

[oherrala]

I also stumbled upon this issue when our monitoring system showed increased memory usage for a monitored process. After starting to scrape the Prometheus endpoint the increase of memory consumption stopped, but the amount of memory consumed didn't drop. Is it possible that the AtomicBucket once allocated doesn't free the memory for the duration of the process?