Using RabbitMQ Cluster Kubernetes Operator step by step but failed to get pod ready

[Darwinarch]

Describe the bug

Hi,
When I followed the instruction online step by step, I could not make the pods ready !

After I described the pod,
kubectl describe po definition-rabbitmq-server-0
I got the following message :

and the log told me that :

my definition.yaml :

my cluster info : only devd02 has PV

 I don't know what's wrong in my setup and I need some help.  Thanks a lot ! 

[ansd]

Hey @Darwinarch,

thanks for reaching out and for the detailed screenshots.

Did you create the file rabbitmq-headless.yml manually?
The selector app: rabbitmq looks suspicious therein.

What should happen instead is that the cluster operator creates a headless service (namely the service/definition-rabbitmq-headless output in your first screenshot) adding a label app.kubernetes.io/name: definition to it.

[Darwinarch]

Hello @ansd ,

I didn't create the rabbitmq-headless.yaml by myself.
Should I create it according to here ?

In addition, service/definition-rabbitmq-headless is generated automatically after I key in kubectl apply -f definition.yaml.
When I described definition-rabbitmq-headless, app.kubernetes.io/name: definition had been labeled as follow:

The difference between definition-rabbitmq-headless and example headless-service.yaml seems like missing to asign port=25672 for cluster-links

Does that setting lead to the pods not ready in my cluster ? And how do I fix it before the service/definition-rabbitmq-headless automatically be started ?

Thanks a lot !

[ansd]

Should I create it according to here ?

No, you shouldn't.

The missing cluster-links port in the headless service is a good catch.
I'm not sure why we didn't include it.
As you correctly pointed out, it is defined in the examples, but not in the operator.
I created #275 to fix that.
Could you try it out with this fix?

My expectation is that it won't fix your issue.

The error message in your screenshot outputs nxdomain (non-existing domain) for the host definition-rabbitmq-server-0.definition-rabbitmq-headless.default.

What do below nslookup statements return in the failing container?
Since I assume that you use a dev environment, what you can try out is to deploy a RabbitMQ instance with the following override:

spec:
  ...
  replicas: 1
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq
              securityContext:
               runAsUser: 0

This allows you to troubleshoot as root user in your container:

kubectl exec pod/definition-rabbitmq-server-0 -ti  -- /bin/bash

# apt update && apt install -y dnsutils
# nslookup kubernetes.default
# nslookup definition-rabbitmq-server-0.definition-rabbitmq-headless.default

What IaaS does your Kubernetes cluster run on?
What Kubernetes version are you using?

[Darwinarch]

Hello @ansd ,

Thanks a lot !

Although I overrode what you mentioned above in my definition.yaml, the
pod/definition-rabbitmq-server-0 rebooted over and over again so that
"kubectl exec pod/definition-rabbitmq-server-0 -ti -- /bin/bash" was not working !

My cluster info is described as follow :

And I followed the instruction online step by step on my docker-desktop yesterday, everything is ok !

but the kubernetes on 【docker-desktop is 1.16.5】 and 【my cluster is 1.18.8】

FYI

Sincerely

[ChunyiLyu]

Hi @Darwinarch

I just tested our operator on a local KinD cluster with k8s v1.18.8 and it's working.

Could you let us know how you provisioned your kubernetes cluster? It's puzzling to me that the k8s version shown in your docker desktop screen shot is 1.16.5, whereas the k8s version shown from your kubectl version is 1.18.8.

I suspect that the failure you are getting is due to some domain/dns setup in your cluster, which is hard for me to debug without knowing how your k8s cluster is provisioned and setup. If you don't have a specific requirement on your k8s environment, I would recommend trying the operator out in a local kinD cluster version 1.16 or above, since that's the setup the team uses and tests on everyday. Thanks!

[Gsantomaggio]

[In additional]
@Darwinarch you can debug the DNS problems by following this guide

using the dns pod:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

Then execute the nslookup:

kubectl exec -i -t dnsutils -- nslookup  definition-rabbitmq-server-0.definition-rabbitmq-headless.default

and the result should be something like:

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   definition-rabbitmq-server-0.definition-rabbitmq-headless.default.svc.cluster.local
Address: 10.244.0.7

Also check the kubernetes dns:

 kubectl exec -i -t dnsutils -- nslookup  kubernetes.default

result:

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

If it does not work, you should check the coredns configuration:

 forward . /etc/resolv.conf
        cache 30

And execute the dns test

you should have, something like:

kubectl exec -ti dnsutils -- cat /etc/resolv.conf

result:

search default.svc.cluster.local svc.cluster.local cluster.local lan
nameserver 10.96.0.10

[Gsantomaggio]

I would close this issue, I don't think it is a bug cc @ChunyiLyu @ansd