Document implications of negative observations for Summaries and Histograms

[beorn7]

tl;dr: If the Observe method is ever called with a negative value, the sum of observations goes down. With the right (un)lucky timing of scrapes, this will lead to a ..._sum time series in Prometheus that doesn't behave like a counter anymore.

Historically, there have been different ideas among the Prometheus developers how to deal with that. See prometheus/prometheus#6669 for a detailed discussion. The conclusion there is that we keep the status quo, which means that in the case above, you cannot meaningfully apply a rate or increase to the ..._sum time series. Had this outcome been anticipated, we would have disallowed negative observations in the first place, but we cannot change that in v1 now (and hopefully, in the future, Prometheus will solve this better). What we have to do now is to document implications. Broadly along the following lines:

• Negative observations are extremely rare in practice.
• In the unlikely case you need them anyway for some reason, that's fine, but note that the ..._sum time series ending up in Prometheus might go down then and thus doesn't behave like a counter anymore. So you cannot use rate or increase on them.
• If you really need both, i.e. negative observations and applying rate or increase to the ..._sum series, there is the work around of creating two Summaries/Histograms, one for the positive observations and one for the negative observations (with inverted sign). You have to combine the results from both with appropriate PromQL expressions.

(And if really anyone needs the latter and has questions about it, then I hereby commit to write a blog post about it.)

[beorn7]

Since this applies to all client libraries, I guess it makes most sense to write some generic documentation in prometheus/docs and link to it from each client library.

[beorn7]

https://prometheus.io/docs/practices/histograms/#count-and-sum-of-observations is the link to use