kube-state-metrics doesn't produce metric for Custom Resources

[mlubanski]

What happened:

I defined configuration for Custom Resource State Metrics for my three CRD's (ReplicationGroup, ClusterInstance, RDSInstance)

After deploying it to cluster where I have kubernetes objects of all above types kube-state-metric is producing me expected crossplane_engine_version metric

But when I deploy same configuration to the cluster where I have only objects of RDSInstance type, crossplane_engine_version metric is missing

What you expected to happen:

Metric should be produced by kube-state-metric even if there are no real Objects of one type/kind listed in configuration below

How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-state-metrics-crd-config
  namespace: crossplane-system
data:
  crd-config.yaml: |
    kind: CustomResourceStateMetrics
    spec:
      resources:
        - groupVersionKind:
            group: cache.aws.crossplane.io
            version: "*"
            kind: ReplicationGroup
          metricNamePrefix: crossplane
          labelsFromPath:
            managedResourceKind: [kind]
            claimKind: [metadata, ownerReferences, "[controller=true]", kind]
            cllaimName: [metadata, labels, crossplane.io/claim-name]
            claimNamespace: [metadata, labels, crossplane.io/claim-namespace]
          metrics:
            - name: "engine_version"
              help: "Engine Version"
              each:
                type: Info
                info:
                  labelsFromPath:
                    engineVersion: [spec, forProvider, engineVersion]
              commonLabels:
                custom_metric: "yes"
        - groupVersionKind:
            group: rds.aws.upbound.io
            version: "*"
            kind: ClusterInstance
          metricNamePrefix: crossplane
          labelsFromPath:
            managedResourceKind: [kind]
            claimKind: [metadata, ownerReferences, "[controller=true]", kind]
            cllaimName: [metadata, labels, crossplane.io/claim-name]
            claimNamespace: [metadata, labels, crossplane.io/claim-namespace]
          metrics:
            - name: "engine_version"
              help: "Engine Version"
              each:
                type: Info
                info:
                  labelsFromPath:
                    engineVersion: [status, atProvider, engineVersion]
              commonLabels:
                custom_metric: "yes"
        - groupVersionKind:
            group: database.aws.crossplane.io
            version: "*"
            kind: RDSInstance
          metricNamePrefix: crossplane
          labelsFromPath:
            managedResourceKind: [kind]
            claimKind: [metadata, ownerReferences, "[controller=true]", kind]
            cllaimName: [metadata, labels, crossplane.io/claim-name]
            claimNamespace: [metadata, labels, crossplane.io/claim-namespace]
          metrics:
            - name: "engine_version"
              help: "Engine Version"
              each:
                type: Info
                info:
                  labelsFromPath:
                    engineVersion: [spec, forProvider, engineVersion]
              commonLabels:
                custom_metric: "yes"

Environment:

• kube-state-metrics version: 2.12.0
• Kubernetes version: v1.28.9
• Cloud provider: AWS
• Other info:

[dgrisonnet]

/assign @rexagod
/triage accepted

[yiyang-shao]

Hi! We encountered the same issue in version 2.12.0 and 2.13.0. It works with version 2.11.0.
After a quick look, we found out that pkg/metrics_store/metrics_writer.go:120(SanitizeHeaders) is causing the problem.
In 2.12.0+, it removes the header from the slice.

// Nullify duplicate headers after the sanitization to not miss out on any new candidates.
if header == lastHeader {
	writer.stores[0].headers = append(writer.stores[0].headers[:i], writer.stores[0].headers[i+1:]...)

	// Do not increment the index, as the next header is now at the current index.
	continue
}

In 2.11.0, it is set to an empty string.

 writer.stores[0].headers[i] = ""

I suppose that this change probably further affects the behavior of the below loop in WriteAll function of MetricsWriter, causing that some metrics are missed in writing.

for i, help := range m.stores[0].headers {
      ...
      for _, s := range m.stores {
	for _, metricFamilies := range s.metrics {
	    _, err := w.Write(metricFamilies[i])
	    ...
	}
      }
}

[greenu]

As suggested in this workaround #2366 (comment), just make help text unique.

[portswigger-tim]

This causes issues with wildcard CR configs too. Moving back to <= 2.11 fixed it for me.

[Duologic]

I just ran into this as well, wildcard matching is essentially broken because of this.