RabbitMQ Node sometimes fails with nxdomain on startup

[mstoetzer]

Describe the bug

Not sure if this is a bug or just a question.
Sometimes I get random restarts of the RabbitMQ pods. When they restart, they produce following log:

This log is from e.g. rabbitmq-server-1:

2022-02-03 16:22:01.153148+00:00 [warn] <0.130.0> cluster_formation.randomized_startup_delay_range.min and cluster_formation.randomized_startup_delay_range.max are deprecated

BOOT FAILED
===========
2022-02-03 16:22:01.563990+00:00 [erro] <0.130.0>
2022-02-03 16:22:01.563990+00:00 [erro] <0.130.0> BOOT FAILED
2022-02-03 16:22:01.563990+00:00 [erro] <0.130.0> ===========
2022-02-03 16:22:01.563990+00:00 [erro] <0.130.0> ERROR: epmd error for host rabbitmq-server-1.rabbitmq-nodes.mynamespace: nxdomain (non-existing domain)
2022-02-03 16:22:01.563990+00:00 [erro] <0.130.0>
ERROR: epmd error for host rabbitmq-server-1.rabbitmq-nodes.mynamespace: nxdomain (non-existing domain)

2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>     supervisor: {local,rabbit_prelaunch_sup}
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>     errorContext: start_error
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>     reason: {epmd_error,"rabbitmq-server-1.rabbitmq-nodes.mynamespace",
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                         nxdomain}
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>     offender: [{pid,undefined},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {id,prelaunch},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {restart_type,transient},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {significant,false},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {shutdown,5000},
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>                {child_type,worker}]
2022-02-03 16:22:02.564897+00:00 [erro] <0.130.0>
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>   crasher:
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>     initial call: application_master:init/4
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>     pid: <0.128.0>
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>     registered_name: []
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>     exception exit: {{shutdown,
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>                          {failed_to_start_child,prelaunch,
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>                              {epmd_error,
2022-02-03 16:22:02.565649+00:00 [erro] <0.128.0>                                  "rabbitmq-server-1.rabbitmq-nodes.mynamespace",

For me it seems that the RabbitMQ pod starts faster than the coreDNS could provide the hostname. For me the message cluster_formation.randomized_startup_delay_range.min and cluster_formation.randomized_startup_delay_range.max are deprecated is a bit irritating.

Version and environment information

• RabbitMQ image: 3.9.10-management
• RabbitMQ Cluster Operator: 1.11.1
• Kubernetes: 1.21.7

Questions:

• Why does rabbitmq-server-1 try to resolve its own hostname via the cluster coreDNS?
• Could it be, that rabbitmq needs some delay to be sure that the hostname is resolvable?
• When cluster_formation.randomized_startup_delay_range.min is deprecated, what will be the replacement option?

[Zerpet]

Hey there 👋 We've seen this ourselved. The cause of the error is nxdomain, which basicaly means that RabbitMQ was unable to resolve the DNS address of itself rabbitmq-server-1.rabbitmq-nodes.mynamespace. If the container starts normally after a restart, it's a timing/race issue, where the RabbitMQ server starts "too soon" before the DNS service updates the records and propagates the changes to the container.

[Zerpet]

IIRC, @ansd worked on RabbitMQ core to address this issue.

[mstoetzer]

@Zerpet Thank you for your answer. The question now is, could this timing issue some how managed with some delay option or something? Do you have any experiences with that?

[Zerpet]

There's a potential fix to the problem here: ansd/rabbitmq-server@dc58985

But it never got merged because we suspected this was an issue specific to kind. Would you be comfortable trying it out in your environmnet?

The steps would be:

• clone https://github.com/rabbitmq/rabbitmq-server and checkout the tag with whatever RabbitMQ Version you’re using.
• cherry-pick / patch above commit.
• in rabbitmq-server directory, you run
  make package-generic-unix && make docker-image

This will create a new docker image that you can deploy on your K8s dev cluster and test whether it fixes your issue.

[mstoetzer]

@Zerpet @ansd Tested the mentioned fix with RabbitMQ 3.9.13. Can confirm that the NXDOMAIN error does not occur anymore. Cool.

Related: vanilla Kubernetes setup via kubeadm, version 1.21.7

[ansd]

Hey @mstoetzer thanks for testing and confirming that it solved your issue.

Could it be, that rabbitmq needs some delay to be sure that the hostname is resolvable?

Yes, that's what rabbitmq/rabbitmq-server#4233 is doing.

When cluster_formation.randomized_startup_delay_range.min is deprecated, what will be the replacement option?

The replacement are Erlang locks. There is no random startup delay anymore. For full context see rabbitmq/rabbitmq-server#3075

[mstoetzer]

@ansd Thank you! Saw that you already pushed this for the 3.10.* release nice!