Refactor the AWS & EKS control-plane controllers to split the kubeconfig secret into two for Cluster Autoscaler #4607
Description
/kind feature
Describe the solution you'd like
Currently, the AWSControlPlane and AWSManagedControlPlane controllers create and manage the cluster kubeconfig secrets ("%s-kubeconfig" in the cluster's namespace). The contents of this secret is a single key-value, with the value being the cluster's kubeconfig in the shape of:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0t...
server: https://XXXXX.us-east-2.eks.amazonaws.com
name: cluster-name
contexts:
- context:
cluster: cluster-3
user: cluster-3-capi-admin
name: cluster-3-capi-admin@cluster-3
current-context: cluster-3-capi-admin@cluster-3
kind: Config
preferences: {}
users:
- name: cluster-3-capi-admin
user:
token: k8s-aws-v1...In the case of EKS clusters, the bearer token embedded in the token field expires quickly and the CAPA controller regularly updates the secret to keep it fresh. However, this shape of config is hard for other controllers, like the cluster autoscaler to consume, and leads to bugs like kubernetes/autoscaler#4784 which can not be easily resolved by those controllers. If they reload the entire config, the user or other critical content may change, if they reload only the token within, then they are disregarding the contents and making unsafe assumptions.
I have discussed this problem with the folks over in SIG API-Machinery and it was suggested to split this secret into two key-values, in this shape:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0t...
server: https://XXXXX.us-east-2.eks.amazonaws.com
name: cluster-name
contexts:
- context:
cluster: cluster-3
user: cluster-3-capi-admin
name: cluster-3-capi-admin@cluster-3
current-context: cluster-3-capi-admin@cluster-3
kind: Config
preferences: {}
users:
- name: cluster-3-capi-admin
user:
tokenFile: ./bearer-tokenAnd a second key-value, "bearer-token" with the contents:
k8s-aws-v1...
Additional context: kubernetes/autoscaler#5951 (comment)
This will allow controllers like the Cluster Autoscaler to mount the secret contents to a path, and it will populate two files. The existing api machinery then should be able to handle the reloading of the 2nd bearer-token file without complications.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Changing the secret value to depend on the 2nd value existing will break any users that consume only the one secret key-value explicitly, in a subPath volume mount. Normal volume mounts should work, as both files will exist relative to each other.