[kube-state-metrics] Fix telemetryPort and Probes for KubeRBACProxy

[Alja9]

What this PR does / why we need it

• Allow users to custom telemetryPort for KubeRBACProxy when it is enabled.

Instead of using hardcode 9091 :

helm-charts/charts/kube-state-metrics/templates/deployment.yaml

Line 64 in 1a04bc9

{{- $telemetryPort := ternary 9091 (.Values.selfMonitor.telemetryPort | default 8081) .Values.kubeRBACProxy.enabled}}

• Fixing probes when KubeRBACProxy is enabled.

Which issue this PR fixes

• When the kube-state-metrics using hostNetwork: true, it may experience a port conflict if port 9091 is already in use.
• Fixing probes when enabling KubeRBACProxy because it uses the port name while the container port is removed if KubeRBACProxy is enabled :

  helm-charts/charts/kube-state-metrics/templates/deployment.yaml

  Lines 162 to 170 in df60c02

  	{{- if eq .Values.kubeRBACProxy.enabled false }}
  	ports:
  	- containerPort: {{ .Values.service.port | default 8080}}
  	name: http
  	{{- if .Values.selfMonitor.enabled }}
  	- containerPort: {{ $telemetryPort }}
  	name: metrics
  	{{- end }}
  	{{- end }}

  
  And use host 127.0.0.1 only for the hostNetwork: true condition, example like this is used in probes:
  

  helm-charts/charts/kube-state-metrics/templates/deployment.yaml

  Lines 175 to 177 in df60c02

  	{{- if .Values.hostNetwork }}
  	host: 127.0.0.1
  	{{- end }}

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [prometheus-couchdb-exporter])

[HaveFun83]

When i try release 6.4.1 with kube-rbac-proxy enabled i got the following errors

  Warning  Unhealthy  7s (x5 over 47s)  kubelet            Liveness probe errored and resulted in unknown state: strconv.Atoi: parsing "http": invalid syntax
  Warning  Unhealthy  2s (x7 over 43s)  kubelet            Readiness probe errored and resulted in unknown state: strconv.Atoi: parsing "metrics": invalid syntax

@Alja9 Can you share your helm values?

[Alja9]

Hi @HaveFun83
Yes, same with you, helm chart 6.4.1.
Since the telemetryPort is hardcoded, I use telemetryPort variable on selfMonitor section even though selfMonitor is disabled to avoid port conflicts.

But I also encountered like that so the problems on the probes, and I ignored the probes.

Warning  Unhealthy  4m7s (x13 over 5m49s)  kubelet            Readiness probe errored and resulted in unknown state: strconv.Atoi: parsing "metrics": invalid syntax
Warning  Unhealthy  52s (x31 over 5m52s)   kubelet            Liveness probe errored and resulted in unknown state: strconv.Atoi: parsing "http": invalid syntax

Okay, I'll try to fix the probes too.

And after I tried again with selfMonitor is enabled, it turned out that telemetry selfMonitor was affected by a conflict between container kube-state-metrics and kube-rbac-proxy-telemetry.

E1107 13:51:43.051361       1 run.go:72] "command failed" err="failed to run groups: failed to listen on secure address: listen tcp :9123: bind: address already in use"
stream closed EOF for monitoring/ksm-2-kube-state-metrics-5bbd6c6dd7-zcl55 (kube-rbac-proxy-telemetry)

I'll try to fix that too.

[HaveFun83]

Thanks a lot for looking into this.
See also #6110
IMHO. For long-term fix we should focus on the auth-filter option #5858 (comment)

I did some basic testing with this option but need more time.

[Alja9]

Yes, I'm looking forward to that feature, while I'm using kube-rbac-proxy .

I've added the newest commit, you can try it again and let me know what you think.
Thank you.

And regarding probes in kube-state-metrics container, it should not use https because the only probes that need to use https are those in the kube-rbac-proxy container. That's what I think and you can see in node-exporter too.