Add an option to allow adding Kubernetes Labels as Prometheus Metric Labels per metric-series

[kostz]

What would you like to be added:
Add an option to tag metrics with kubernetes labels

Why is this needed:
The case I'm working on involves otel-collector scraping prometheus metrics from KSM and exporting them into the commercial backend. The thing is that commercial backend charges for metric cardinality, so reporting metrics with such labels as pod_name and then do joins would cost a lot.

One of the possibility to avoid this excessive costs is to shrink cardinality by aggregating before exporting metrics (well, with acceptable loosing of visibility of keeping data for each pod) and it looks very promising to have this labelling on KSM and then remove extra labels (like pod_name, uid, etc) on otel-collector's processor.

Describe the solution you'd like
Have a feature toggle similar to metric-allowlist which will not add labels to kube-pod-labels metrics, but add labels for related metric timeseries, like kube_pod_container_resource_requests

Additional context
Follow up for #1758

[dgrisonnet]

My concerns from #1758 (comment) still applies.

You could today remove labels via the otel-collector relabeling config but more complex aggregation similar to what Prometheus has with recording rules is not possible: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md#unsupported-features. Maybe there is a way to do post-processing in Otel, but that is out of kube-state-metrics' scope.

[dgrisonnet]

This is what you are most likely looking for: open-telemetry/opentelemetry-collector-contrib#4968

[kostz]

@dgrisonnet , thanks for quick reply, my bad of using the 'aggregation' term, that may lead to confusion about caching something and doing a real aggregation in KSM, that's not I'm asking for, I'm asking just about having an option to tag time series points with tags taking them from pod specs.

In my case real aggregation will be done in that commercial metrics backend, but having the pod labels from KSM (to aggregate by) and removing id-labels (in otel-collector processor) will dramatically decrease cardinality of time series and saves a lot.

Honestly I didn't get a point about enormous memory footprint you're projecting for this feature, KSM will still deliver metrics almost 1:1, but adding additional (limited) tags taking them from pod specs

Also this feature may affect not the whole set of metrics, but just some of them (well, we're using less then 10% IIRC)

[kostz]

Also I would like to say that cadvisor already has this feature out of the box to bypass container labels to time series with CLI argument whitelisted_container_labels, and that works pretty well - https://github.com/google/cadvisor/blob/master/cmd/cadvisor.go#L66

So @dgrisonnet, why not to have the same in KSM?

[logicalhan]

/triage accepted
/assign @dgrisonnet

[dgrisonnet]

Honestly I didn't get a point about enormous memory footprint you're projecting for this feature, KSM will still deliver metrics almost 1:1, but adding additional (limited) tags taking them from pod specs

All these new label will have to be stored in both ksm cache and Prometheus timeserie database. When you have 100 timeseries it is not that much, but when you have millions it is a lot of memory you are wasting.

In my case real aggregation will be done in that commercial metrics backend, but having the pod labels from KSM (to aggregate by) and removing id-labels (in otel-collector processor) will dramatically decrease cardinality of time series and saves a lot.

Could you perhaps share an example of what the metrics will undergo? It seems to me that you want some kind of middleware between ksm and your monitoring backend that would aggregate the metric by label before storing them in order to reduce the cost no?

Also I would like to say that cadvisor already has this feature out of the box to bypass container labels to time series with CLI argument whitelisted_container_labels, and that works pretty well - https://github.com/google/cadvisor/blob/master/cmd/cadvisor.go#L66

It could make sense for cAdvisor since it doesn't have dedicated _labels metrics like ksm, but I don't think that is the wisest choice. For sure it might be the most convenient but it is definitely not the most efficient. Metrics should be as minimal as possible and serve only one purpose.

@mrueg do you have any thought on that?

[framegrace]

I have urgent need of this feature.
I think an arbitrary way to categorize the metrics will be very useful, so they appears on prometheus queries.

It allows to split metrics by internal classifications, other than cluster. (To send alerts to correct teams, split metrics in multi-tenant clusters ....). I really think it's a must feature.

I understand the implications of adding labels to the metrics, but I maybe adding just one extra optional label on each metric would be acceptable.
The label can be called "classifier" or something similar, which can be assigned to an arbitrary label of the resource and it can be made optional, so current installs are not affected.

But well, at the end, either need that or a way yo have multiple kube-state-metric processes handling different resources selected by label so I can process this situation later on prometheus rules.

My current situation is that I need to split metrics by "project_id" to thanos, so each customer can see it's own metrics without seeing the ones for the rest. And I'm currently out on the water trying to implement this with current options.

[dgrisonnet]

It allows to split metrics by internal classifications, other than cluster. (To send alerts to correct teams, split metrics in multi-tenant clusters ....). I really think it's a must feature.

This can already be achieve by doing joins on kube_*_labels metrics to surface the labels that you wish to see in your alerts.

My current situation is that I need to split metrics by "project_id" to thanos, so each customer can see it's own metrics without seeing the ones for the rest. And I'm currently out on the water trying to implement this with current options.

To achieve this kind of tenancy, we personally use https://github.com/prometheus-community/prom-label-proxy in OpenShift and it does the work pretty well.