change the kubelet service crash loop behavior

[neolit123]

over time we have seen a number of complains related to the crash loop of the kubelet service in the DEB/RPMs. when the kubelet is installed, the service is enabled but it would fail because it's missing its config.yaml (KubeletConfiguration), unless something like kubeadm creates one for it.

this has caused problems for:

• Windows support
• supporting other service manager like OpenRC

after a discussion of the kubeadm office hours of 10.06.2020 we've agreed that it might be a good idea to change this behavior and keep the service disabled by default. but this would require changes in both kubeadm and the kubelet systemd specs.

the idea we are thinking of is the following:

• modify kubeadm to always enable the service if its not enabled on kubeadm init/join runtime.
  note that, currently kubeadm just has a preflight check that fails if the service is not enabled and instructs the user how to enable it manually.
• modify the kubelet systemd service files in the kubernetes/release repository to have the kubelet service disabled by default. this change will require a release note with "action-required" as non-kubeadm users would have to manually enable it (e.g. using: "sysctl enable kubelet").

/kind feature
/priority important-longterm

[neolit123]

the kubeadm change is doable for 1.19, but arguably the k/release change needs wider agreement.

[neolit123]

xref kubernetes/release#1352

[neolit123]

cc @rosti
so i think there is a limitation of our release process as i mentioned here:
kubernetes/release#1352

which means that the PR might be much better than the above proposal.

[BenTheElder]

I thought kubeadm already had code related to doing this https://github.com/kubernetes/kubernetes/blob/875f31e98878fd199a76fd0ba2465d14558788cd/cmd/kubeadm/app/phases/kubelet/kubelet.go#L38

[neolit123]

it only starts and restarts the service but does not manage "enable" status.
it does tell the user how to enable, though.

[rosti]

The necessary modifications here are:

1. Remove the kubelet service running check (from here)
2. Enable the service during init and join. One way is to add the enabling code where the TryStartKubelet calls are. A safer and somewhat backwards compatible alternative is to add new kubelet enable phases to init and join that would just enable permanently the kubelet service after all of the rest of the init/join process has been successfully completed.
3. Disable the kubelet service during kubeadm reset. Again, this can be sitting next to the TryStopKubelet call or be in a separate reset phase.

As said previously. It doesn't hurt for us to implement this. It's not that big of a change on our side. The problem is what to do with the packages where the kubelet service continues to be automatically enabled and crash looping (since it doesn't have a config file).

[neolit123]

your proposed plan for kubeadm seems fine to me.

The problem is what to do with the packages where the kubelet service continues to be automatically enabled and crash looping (since it doesn't have a config file).

so i think ideally we should be making this change only for the latest MINOR.
but in kubernetes/release#1352 we are also discussing the fact that currently such changes are applied to all PATCH release of k8s packages.

currently the kubeadm package has the kubelet package as a dependency (there were discussions to change this too), which supposedly installs the same versions for both package for most users.

there could be users that are bypassing that and installing e.g. kubeadm 1.x and kubelet 1.x-1, due to factor X, and this is a supported skew. for such a skew the old kubelet service may be enabled by default (crashloop) but the new kubeadm could be managing "enable" already.

testing something like systemctl enable kubelet on a service that is already enabled doesn't have an exit status != 0, for windows Set-Service ... -StartupType.. should not return errors. no idea about OpenRC. in any case we may have to call the potential InitSystem#ServiceEnable() without catching its errors, but later fail on InitSystem#ServiceStart().

so overall even if a kubeadm binary encounters an older kubelet service, for the crashloop problem in particular this should be fine, unless i'm missing something.

however for the kubelet flag removal and instance specific CC problem, the kubeadm 1.x and kubelet 1.x-1 skew might bite people that are passing flags to a kublet that no longer supports flags at all.
i can see us removing kubeletExtraArgs from the kubeadm API at some point. but that's a separate topic.

[xlgao-zju]

Tested systemctl enable kubelet.service on a service that is already enabled, and get exit code = 0.

I think @rosti 's plan is fine, and I'd like to help with this.

[xlgao-zju]

/assign