Huge amount of reconciles #250
Description
As reported here in RabbitMQ Slack, there are scenarios that result in huge amount of reconciles (triggered by deleting the messaging-topology-operator pod).
This script seems to reproduce the issue "most of the time":
Trigger reconciles
#!/usr/bin/env bash
set -euxo pipefail
kind create cluster --image kindest/node:v1.21.1
kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/download/v1.9.0/cluster-operator.yml
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
sleep 10
kubectl -n cert-manager wait --for=condition=Ready pods -l "app.kubernetes.io/instance=cert-manager" --timeout=5m
kubectl apply -f https://github.com/rabbitmq/messaging-topology-operator/releases/download/v1.2.0/messaging-topology-operator-with-certmanager.yaml
kubectl rabbitmq create r1
kubectl wait --for=condition=AllReplicasReady rmq/r1 --timeout=5m
exchange=$(cat << EOF
---
apiVersion: rabbitmq.com/v1beta1
kind: Exchange
metadata:
name: exchange-{{index}}
spec:
name: exchange-{{index}}
vhost: "/"
type: direct
autoDelete: false
durable: true
rabbitmqClusterReference:
name: r1
EOF
)
for i in $(seq 1 150); do
echo "$exchange" | sed "s/{{index}}/$i/" | kubectl apply -f-
done
sleep 10
kubectl -n rabbitmq-system delete pod -l "app.kubernetes.io/name=messaging-topology-operator"
sleep 5
kubectl -n rabbitmq-system wait --for=condition=Ready pods -l "app.kubernetes.io/name=messaging-topology-operator" --timeout=2m
# Now we see lots of reconciles:
kubectl -n rabbitmq-system logs -l "app.kubernetes.io/name=messaging-topology-operator" -fAbove script uses exchanges, but the same issue occurs also for other objects / controllers (including queues and vhosts).
The root cause is that the controllers update the LastTransitionTime in the status conditions although there were no transitions. The field is supposed to be updated only if condition transitions from one status to another.