Debug RabbitMQ

This guide is a collection of information which can be useful for root cause analyses after an issue occured.

Debug logging

You can temporarily enable debug logging for all RabbitMQ pods by running the following on a single pod:

nodes=($(rabbitmqctl cluster_status --formatter json | grep --only-matching "disk_nodes\":\[\"[a-z@0-9.\",-]*\]" | cut -f2 -d":" | sed -e "s/^\[\(.*\)\]$/\1/" -e "s/[\"\r]//g" -e "s/,/ /g"))
for node in "${nodes[@]}"; do
    rabbitmqctl -n "$node" set_log_level debug
done

Note that the log level will be reset after a restart of the pod.

Inspect unconsumed quorum queue messages

If quorum queues start accumulating messages, this indicates an issue between the communication of two services. Either the consumer fails to retrieve messages, the publisher failed to detect that the consumer was intentionally removed or the publisher is creating messages at a faster rate than the consumer can process.

To get a JSON object of the unconsumed messages of a queue, you can run the following script:

./tools/get-rabbitmq-messages.sh <amqpserver> <queue_name> -a > /tmp/messages.json

Note that this script is tailored specifically to parse messages sent by oslo.messaging.

The methods the publisher intends to call are usually defined inside files called rpcapi.py of the OpenStack repositories where each service defines its own API.

rabbitmqctl eval

Attention

Never use rabbitmqctl eval commands in productive environments without sufficient testing.

rabbitmqctl eval can be used to execute exported methods of the RabbitMQ codebase. This is very useful to get information about existing data which can not be retrieved by existing RabbitMQ commands, or to find ways to troubleshoot a cluster.

For example, if you want to list all records of the table in which durable queues are defined, you would find out that there is the method get_all_durable inside the module rabbit_db_queue. You can run that method using

rabbitmqctl eval 'rabbit_db_queue:get_all_durable().'

You could also combine the return value with other erlang expressions.