Avoid hard-coded values for API server address #288
Conversation
Avoid hard-coded value of 127.0.0.1 and 6443, as the API server can listen on another address when used outside of k3s. When used from another go program using the `chart` package, host and ports should be passed as argument. When used standalone, host and port should be derived from the kubeconfig and/or master-url options.
| return "", "", err | ||
| } | ||
|
|
||
| return u.Hostname(), u.Port(), nil |
There was a problem hiding this comment.
You cannot assume that the in-cluster apiserver port is the same as the actual port on the host. For example, on both K3s and RKE2, the port for the in-cluster kubernetes service is 443, and that's what you'll see here if you look at the client config - but the apiserver is actually listening on 6443 on the server nodes.
You need to understand the difference between service ports, and service endpoints. Bootstrap charts need to access the local apiserver endpoint directly, as kube-proxy and the rest of the CNI cannot be relied on to be functional yet.
root@systemd-node-001:/# kubectl get service kubernetes -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 21h <none>
root@systemd-node-001:/# kubectl get endpointslice kubernetes -o wide
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
kubernetes IPv4 6443 172.17.0.4 21hThere was a problem hiding this comment.
Yes, I understood that.
I tried the patch with minikube and k3s by using helm-controller outside the cluster. In that case it worked since the kube-config file is set with direct api endpoint and not the in-cluster service.
I forgot to try the in-cluster image. I suppose in that case the code will use the in-cluster service and fail.
I will try and report back.
There was a problem hiding this comment.
Yes, doing this the "right way" such that it will work in all possible environments is non-trivial, which is why I opted to just fix it the easy way in K3s.
There was a problem hiding this comment.
I couldn’t find a use-case where the helm controller can connect to the API but not the helm bootstrap chart pod.
If the helm controller is using the in-cluster api and has access to the custom resources, then the pod running the bootstrap chart will be able to connect as well to the in-cluster API.
If the helm controller is configured to connect to the in-cluster API and the in-cluster API is not available, then the helm controller will never see the custom resource and will never start any pod anyways.
So using the in-cluster API for bootstrap charts when helm controller is using in-cluster API shouldn’t be a problem?
Avoid hard-coded value of 127.0.0.1 and 6443, as the API server can listen on another address when used outside of k3s.
When used from another go program using the
chartpackage, host and ports should be passed as argument.When used standalone, host and port should be derived from the kubeconfig and/or master-url options.
I’m not sure how version number shoud be changed, since this breaks API for chart package.
It could also be relevant to add tests?
Related to k3s-io/k3s#12947 and #287