Restore RabbitMQ cluster¶

If you have some RabbitMQ cluster issues, this guide can help you to recover them.

This guide doesn’t cover all cases but may help to get a first look.

Useful commands:

Check cluster status:

rabbitmqctl cluster_status

Check the number of disk nodes and running nodes. This should fit to the number of pods.

Rebuild everything¶

If all members of your cluster are down and no RabbitMQ communication is working between the nodes and the clients using rabbit, this is the easiest approach.

Warning

BUT: It comes with loss of all messages, currently stored on the RabbitMQ Cluster!!

Delete all pvc of the RabbitMQ (e.g. data-cinder-yaook-cinder-mq-0.)
Force delete all Pods of the mq statefulset. As you delete all data, you also don’t need to gracefully shut down the Pods.
Wait till the statefulset recreates the Pods and check if they are healthy.
Wait till the infra-operator reconciles the corresponding AMQPUser. You can immediately trigger the reconcile by restarting the infra-operator or by touching the AMQPUser resource (e.g., by adding and removing the state.yaook.cloud/pause annotation).
Restart all clients of this RabbitMQ, so they recreate required queues and exchanges.

Rebuild from healthy node¶

As long as you have one healthy node in the cluster, you can try to rebuild from that one.

First approach: Restart pods and autojoin¶

Restart the remaining pods:

Force delete all bad pods.
Wait and check if they come up and join the cluster again.

But if you have issues restarting RabbitMQ, this most probably don’t help.

Second approach: Remove nodes from the cluster:¶

Instead of restarting the Pods and automatic join remove the node from the MQ cluster and let it join again.

Stop the broken service:
```
rabbitmqctl -n <NODE> stop_app
```
This can either be done on the to be removed pod itself or via another pod of the Cluster. <NODE> is something like rabbit@cinder-yaook-cinder-mq-0.cinder-yaook-cinder-mq-rdy.yaook.svc.cluster.local
Drop the node from the cluster.

Run on the pod of the healthy node:
```
rabbitmqctl forget_cluster_node <NODE>
```
On the to be removed pod, reset the node.

This can be done by deleting the PVC of the pod and restart it,

or via rabbitmqctl
```
rabbitmqctl reset
```
Join the node again to the cluster.

If you deleted the PVC and restarted the pod, this should be done automatically.

When you have used reset run:
```
rabbitmqctl join_cluster <cluster>
rabbitmqctl start_app
```
<cluster> can be the name of one healthy node, e.g. rabbit@cinder-yaook-cinder-mq-1.cinder-yaook-cinder-mq-rdy.yaook.svc.cluster.local

Solve Pod stuck in failed startup probe¶

It can happen that a rabbitmq Pod fails to become ready. Besides checking the pod logs, also check the pod description if the startup probe fails.

Yaook has a startup probe for MQ pods, that checks if the Node has joined the cluster. If joining the cluster fails, the node comes up as standalone. If you/ yaook don’t see this, you can end up with separate standalone nodes and no one will notice (till some issues occur.)

This can happen, when the MQ PVC got deleted and the pod starts again. The cluster then still knows the old Node with same name and won’t allow the new one to join.

To resolve this, follow the steps above on Remove nodes from the cluster.

Solve failed readiness probes¶

To find out why the readiness probe is failing, run the following:

rabbitmq-diagnostics -q check_running
rabbitmq-diagnostics -q check_alarms

If the node is not running, inspect the logs to find out why the pod terminated. For single node disk alarms, the readinessProbe automatically terminates the node to not make the entire cluster unresponsive.

If the failure was caused by a memory alarm, you might have to increase the messageQueue limits inside messageQueue.resources.limits.memory.

For disk alarms, first check for accumulating messages inside quorum queues if the queue size limit is disabled: Fix broken queues

Otherwise you either have to adjust the storageSize of your AMQPServer, or you encountered a bug. To figure out which files which take up the most space, run:

du -ah /var/lib/rabbitmq/mnesia/ | sort -hr | head

Compare the file sizes to the limits configured by the policies: rabbitmqctl list_policies

Each directory inside /var/lib/rabbitmq/mnesia/$RABBITMQ_NODE_NAME/quorum/$RABBITMQ_NODE_NAME corresponds to a quorum queue. To know which quorum queue it belongs to, you can read the file config inside that directory:

grep -A1 "friendly_name" config

Also ensure that the WAL file does not grow too large and there is no issue cleaning up old WAL files. For more information see the RabbitMQ documentation.

If you encountered a bug, the quickest way to make the cluster operational again is likely to delete files, recreate queues or even recreate the entire AMQPServer.