[RFC] Support for out-of-service taint

[k-keiichi-rh]

kubernetes/enhancements#1116 is merged into k8s 1.24 as a beta feature.
And it will become a stable feature in k8s 1.26.

This KEP introduces a new "out-of-service" taint that ensures pods on the node will be forcefully
deleted and the volume detach operations on the node will happen immediately.
It means the deleted pods can recover quickly on different nodes.

This feature will have a good impact on the existing remediation mechanisms.
But there is a restriction that the user has to confirm the node is in 'NotReady' status
and the node is shutdown or in a non-recoverable state.

If the above conditions are not satisfied, pods on the unhealthy node are not deleted
and the volumeAttachment objects are not also removed.

There are several options to confirm the above conditions in SNR:

1. Fencing the node by using any management interface (e.g. IPMI etc.)
   => But SNR doesn't have the prerequisite according to https://www.medik8s.io/remediation/poison-pill/faq/#how-poison-pill-is-different-from-other-solutions
2. Intentional kernel panic
   => softdog supports the "soft_panic" option.
3. Soft power off
   => We can't trust soft power off when the node is unhealthy.
4. Other options?

Any comments are very welcome.
And if my understanding is wrong, please correct me.

For reference, the following flow is how NHC and SNR work:

1. When NHC detects a node failure, it creates a PoisonPillRemediation(PPR) object for the remediation.
2. SNR starts to remediate the node according to the PPR object.
3. SNR marks the unhealthy node as "SchedulingDisabled"
4. The unhealthy node reboots(*1)
   => We have three tiers(1. hardware watchdog, 2. softdog, 3. software reboot).
6. SNR waits for 180 seconds(default) before it starts deleting affected workloads like pods and volumeAttachment objects.
7. SNR takes a action according to the PPR object. We have "NodeDeletion" and "ResourceDeletion" as remediation
8. SNR deletes the Node Object(in case of the "NodeDeletion") or free the affected resources(in case of "ResourceDeletion").
9. After recovering the node, SNR unmarks the "SchedulingDisabled" flag from the unhealthy node.
   SNR restores the deleted Node Object(in case of the "NodeDeletion")

(*1)
SNR on a node will trigger reboot if
  1. it sees a remediation CR for itself
  2. or it can't connect to the API server AND
   2a) at least one peer SNR reports a remediation CR exists
   2b) or it fails to ask peers (in this case no one will delete any resources though, peers don't know about the problem)

It will NOT reboot if it can't connect to the API server AND
  1. at least one peer SNR reports there is NO remediation CR
  2. more than 50% of the peers which were contacted so far report API server issues as well (probably means the API server itself has an issue itself, or t\
here is a general network issue)

[slintes]

1. Fencing the node by using any management interface (e.g. IPMI etc.)
   => But SNR doesn't have the prerequisite according to https://www.medik8s.io/remediation/poison-pill/faq/#how-poison-pill-is-different-from-other-solutions

Exactly. Actually this is a key feature of PP, that we don't need a management interface, so I don't think we want to change this. If you want to use a management interface, you need to use another remediator (e.g. from metal3, see https://github.com/metal3-io/metal3-docs/blob/main/design/cluster-api-provider-metal3/capm3-remediation-controller-proposal.md)

2. Intentional kernel panic
   => softdog supports the "soft_panic" option.

At the moment we also try to recover the node, so reboot only. But I think we could also introduce panicking as a new remediation strategy.
How reliable is the softdog though?

3. Soft power off
   => We can't trust soft power off when the node is unhealthy.

agree

4. Other options?

[k-keiichi-rh]

I looked into the sources and documents for SNR. I misunderstood.
I found out that the softdog is not reliable and SNR recommends using hardware watchdog.
Is this right?

At the moment we also try to recover the node, so reboot only. But I think we could also introduce panicking as a new remediation strategy.
How reliable is the softdog though?

I think it's reliable and I can't imagine what kind of case it fails to reboot or panic.
But I don’t know what will happen. So the hardware watchdog is more reliable than softdog.
So I think it's good as one of the options, not main one...

[beekhof]

HA is a continuum of risk...
IPMI/DRAC >> hardware watchdog >> software watchdog >> software reboot

None give you absolute guarantee that the node is safe, but if you don't or can't have the more preferable options, then anything is better than nothing as long as you understand the failure scenarios, their chances of happening, and impact if they do

I offer both http://www.beekhof.net/blog/2019/savaged-by-softdog which explains how softdog might break, and my subsequent testing in which I was unable to get softdog to do the wrong thing.

[beekhof]

@slintes

At the moment we also try to recover the node, so reboot only.

Unless I'm missing the context, I don't think thats true.
The reboot was supposed to be in addition to panic triggered by a watchdog (assuming a watchdog is present).

However we could have the operator also try to explicitly panic the box if that is what you were suggesting.

[slintes]

@slintes

At the moment we also try to recover the node, so reboot only.

Unless I'm missing the context, I don't think thats true. The reboot was supposed to be in addition to panic triggered by a watchdog (assuming a watchdog is present).

I didn't mean the fallback software reboot. The watchdog also does a reboot, not a shutdown/panic?

However we could have the operator also try to explicitly panic the box if that is what you were suggesting.

Yes, in case a reboot doesn't work well with the out-of-service taint, that might be an option. But I'm unsure if that's even possible to implement, because the remediation strategy is part of the remediation CR, and not a global option in the config CR. So we would need to reconfigure the watchdog on demand. But the agent on the broken node might not even be aware of the remediation CR, so it would not know that it needs to panic instead of reboot... 🤔

But maybe I'm looking for too much guarantees 🤷🏼‍♂️

[beekhof]

Softdog can do either
https://elixir.bootlin.com/linux/latest/source/drivers/watchdog/softdog.c#L87

I was under the impression panic was the default. Maybe this was the case historically, but this seems not to be true today.

@mshitrit We will need to make sure SNR configures this option.

[k-keiichi-rh]

Our community team suggested me that SetNodeCondtion() may help us to use the out-of-service taint.

Now there is a restriction for the taint that user intervention is required to detect node shutdown/failure cases and the user has to confirm that the node is shutdown or in a non-recoverable state.

As we discussed in #16, SNR resets the unhealthy node by the hardware watchdog and the NoExec taint can ensure that nothing runs on the node when it returns after resetting.
This situation doesn't mean the node is shutdown, but we can say it's close to the "the node is shutdown" that means we can safely evict workloads from the unhealthy node.
If we can change the node condition to the "NotReady" by the SetNodeCondtion(), we can use this taint without changing the current mechanism of SNR drastically.

As far as I checked the node condition check for the taint, the kube-controller-manager verifies that the node is NotReady. It means the IsNodeReady(the node) returns false.

I will try to confirm whether the above scenario works fine or not.
In my opinion, It doesn't seem like the moderate way and we have to consider the side effect when changing the node condition intentionally. But I think that this way will works well as expected.
This is the sample code for testing k-keiichi-rh@990884d.

Please let me know if you have any comments and feedback.

[beekhof]

Thinking out loud...

If we do the normal SNR thing, and apply the out-of-service taint once the timer expires, then one of two things will happen when we apply the taint:

1. the node is still unable to contact the control plane (either because it's still booting or is permanently failed)

The NotReady condition remains present, and the taint causes affected workloads to be forgotten, and moved elsewhere

2. the node is already back and healthy (able to talk to the control plane)

The NotReady condition will have been removed, and the taint ignored.
However the node will also be healthy enough to report its current state (including that nothing is mounted/running) to the control plane and affected workloads will get started elsewhere.

Am I missing something? Seems like we get the desired behaviour either way.

[k-keiichi-rh]

Am I missing something? Seems like we get the desired behaviour either way.

Thank you for correcting me. That makes sense to me. I think we can get the desired behavior either way.
As far as I tested, I got the expcted result as you mentioned.

Please let me confirm whether my understanding is correct or not.

If the out-of-service taint is ready in k8s, we can provide the new remediation strategy in the following:

[ The new remediation strategy for the out-of-service taint ]

1. Worker fails
2. Surviving workers start their timer
3. The "medik8s.io/remediation=nodefencing:NoExecute" taint is added to the Worker
   => This is implemented in https://github.com/medik8s/self-node-remediation/pull/21/files
4. Worker’s watchdog triggers a reboot
5. Timer of the surviving peers expires
6. Surviving peers clean up the worker
   => The "node.kubernetes.io/out-of-service=nodeshutdown:NoExecute" taint is added to the Worker
7. Worker becomes healthy again
8. Worker contacts the control plane
9. Control plane to the worker to run the same pods

As for Step 3, the pod deletion equivalent to "kubectl delte pods" is executed.
But the reliability issues like data corruption won't be caused.
Because pods with PVC remain in the "Terminating status" and VolumeAttachment objects are not released.
These clean up is executed after rebooting with the NoExec taint.
So the main purpose of this taint is that ensures nothing runs on the node after rebooting.

As for Step 5, there are two cases:

1. The node is in the "Not Ready" status
   The out-of-service taint deletes the pods and the VolumeAttachement objects as explained in [RFE] Ensure any pods are not running on unhealthy node after rebooting #16 (comment)
   But the k8s community has a requirement for this taint that the unhealthy node should be completely isolated before adding the taint.
   So, it's better to provide a docuement to explain that users need to make sure the node was shutdown or in a non-recoverable state when using this strategy and the node is remediated

2. The node is not in the "Not Ready" status
   If the node can contact the control plane, the pods with the stateful workloads will be deleted as tested.
   But we can't trust the unhealthy node, so it's better to isolate the unhealthy node as soon as possible user find the system failures.

There is no perfect way to meet user's requirements.
So, if users want to completely isolate unhealthy nodes, another remediator with IPMI/BMC access should be recommended.

[mshitrit]

[ The new remediation strategy for the out-of-service taint ]

I've noticed that you haven't mentioned the removal of the out of service taint.
Is it currently out of scope ?

2. The node is not in the "Not Ready" status
   If the node can contact the control plane, the pods with the stateful workloads will be deleted as tested.

IIUC this use case should be prevented by the "medik8s.io/remediation=nodefencing:NoExecute" taint.
Since we place this taint early on (first thing when the unhealthy node is detected) it will exit before and after the Node reboots.
So after the reboot even if the node is healthy I don't expect any workload on it due to this taint.

[k-keiichi-rh]

[ The new remediation strategy for the out-of-service taint ]

I've noticed that you haven't mentioned the removal of the out of service taint. Is it currently out of scope ?

I was focusing adding the taint and I didn't care about removing the taint.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown#handle-the-return-of-shutdown-node

 In the future, we can enhance the Pod GC Controller to automatically remove the out-of-service taint, after verifying that no pods are attached to the node, and no volume attachments on the node.

According to the above sentence, I think we can remove the taint if we can confirm nothing runs on the node.
What do you think?

[mshitrit]

According to the above sentence, I think we can remove the taint if we can confirm nothing runs on the node.
What do you think?

I Agree

[k-keiichi-rh]

Now the SNR enables a remediation based on the out-of-service taint.
So I close this issue.