Questions about Kubernetes upgrades when K8S is running on AKS

[sebastianlung]

Describe the bug

We are running RabbitMQ within an Azure Kubernetes service.
Since we deployed RMQ, we are having issues, while we do an update of the k8s version.

New nodes are deployed but the old nodes can't be drained completly because the rabbitmq-cluster-server-0/rabbitmq-cluster-server-1 pod is in "Terminating" state but never completly terminated. The only solution I have so far is to force the pod to terminate with:
kubectl delete pod --force --grace-period=0 rabbitmq-cluster-server-0

Then we have to manually delete the pvc so they can be attatched to a new node.

Reproduction steps

1. Deploy RMQ to a AKS
2. Upgrade the AKS version
3. RMQ Pods are not beeing terminated on the old nodes

Expected behavior

All nodes can be drained and the RMQ pods are stopping correctly and no manual interference regarding pvc.

Additional context

No response

[michaelklishin]

@sebastianlung you have filed this as an issue in the documentation repo.

This issue as filed does not meet our team's criteria, specifically "Deploy to AKS" is not a reproduction step, and neither are steps 2 and 3.

Pretty much all the key K8S deployment details that an Operator maintainer would like to know are missing. We do not guess in this community.

So please provide those details, or at least evidence and details on why you think that something in this specific Operator is directly responsible for the state of the pods. Or feel free to investigate this behavior on your own. This is an open source project after all.

[sebastianlung]

Sorry for that, I'll try to provide more details

[MirahImage]

There are a number of things which can cause a node to fail to shut down. The most common tends to be that there is a quorum queue with only the minimum number of online replicas (e.g. in a 3 node cluster, having only 2 replicas), meaning shutting down a node is prevented because it is quorum critical.

Either way, we will need more details if we are to understand and investigate this issue. How many nodes are in your cluster? What type of workload are you running (quorum queues? streams? purely classic queues?)? What version of RabbitMQ are you running?

[sebastianlung]

Hi, thanks for the hint. This happend on our DEV environment, which only has two RabbitMQ nodes, but uses quorum queues. I'll scale up to 3 or 5 nodes, reconnect to the queues and check if the issue persists.

[michaelklishin]

Two node clusters is a configuration explicitly recommended against. A quorum queue with two replicas can tolerate failure of zero replicas per Raft (and common sense) definition of a quorum (majority).

[sebastianlung]

You are absolutly right. In my defense, I didn't set it up myself 😅 I was just stuck with the Update. After solving the issue the update went through.

[michaelklishin]

Sorry, I wasn't trying to shame you, simply state a documented limitation of RabbitMQ and nearly all Raft-based systems I have seen :)