Configure RabbitMQ¶

Reduce startup time¶

Before startup, each RabbitMQ pod executes a sleep command to prevent negative DNS caching results which can cause problems with inter node communication for stream queues during restarts of RabbitMQ.

If you are working in a productive environment, this should only be done if the the DNS cache TTL of the Kubernetes DNS server is changed first. Afterwards, the sleep duration can be overwritten by setting messageQueue.dnsCacheTTL inside the OpenStack manifest accordingly.

Inside development clusters it is fine to set the value 0 without adjusting the DNS server, just expect to encounter some error messages concerning stream queues.

Tuning RabbitMQ CPU and memory usage¶

Important

By default, RabbitMQ and Erlang assume that they do not share CPU and memory with other processes on the host. Because of this, we recommend that you tune them down with one of the following methods.

In Yaook, the following RabbitMQ and Erlang settings related to CPU and memory usage can be tuned:

The number of scheduler threads (+S Erlang flag)
The size of the default worker pool (default_worker_pool_size)
The scheduler busy wait threshold (+sbwt Erlang flag)
The total available memory (total_memory_available_override_value)

Except for the busy wait threshold, the most basic way to adjust these values is by setting CPU and memory resource requests and/or limits for the AMQPServer, like you do with every other Kubernetes workload. For example:

kind: AMQPServer
spec:
    resources:
        rabbitmq:
            requests:
                cpu: 8500m
            limits:
                memory: 4Gi

The Yaook Operator will then choose appropriate values based on the resource specifications that you provide. These settings should work well in most of the cases:

The default number of scheduler threads and the size of the default worker pool will be calculated based on the CPU request (or CPU limit, but CPU request takes precedence). The value is rounded down to a full CPU, but will be at least one. Example: a CPU request of 8500m will configure 8 scheduler threads and set the default worker pool size to 8.
The config option total_memory_available_override_value is calculated by using the memory limit minus some headroom to avoid OOM scenarios.

Note

In Yaook, the CPU resource request value takes precedence over the CPU resource limit when the default config parameters are generated. This means that the Erlang process of RabbitMQ will only use as much cores as defined in the CPU requests, even if you have a much higher limit set.

If necessary, the number of scheduler threads can be configured explicitly with spec.schedulerThreads and the default worker pool size can be configured explicitly with spec.defaultWorkerPoolSize.

The scheduler busy wait threshold can be tuned with spec.schedulerBusyWaitThreshold. It currently defaults to medium, but unless you are running dedicated RabbitMQ nodes, you should use none to save CPU resources and avoid potential noisy neighbor issues.

Examples¶

Basic configuration of the message queue for the Cinder service, using the CinderDeployment top-level resource:

kind: CinderDeployment
spec:
    messageQueue:
        resources:
            rabbitmq:
                requests:
                    cpu: "4"
                limits:
                    memory: 2Gi
        schedulerBusyWaitThreshold: none
    ...

Advanced configuration using the AMQPServer subresource directly:

kind: AMQPServer
spec:
    resources:
        rabbitmq:
            requests:
                cpu: 8500m
            limits:
                cpu: "12"
                memory: 4Gi
    schedulerBusyWaitThreshold: none
    schedulerThreads: 6        # overrides the calculated value of 8 (based on CPU request)
    defaultWorkerPoolSize: 10  # overrides the calculated value of 8 (based on CPU request)
    ...

Override images¶

If you want to use custom images for an AMQPServer, do so by adding a dictionary to .spec.imageOverrides with the pinned versions as keys and the image overrides as values. For upgrades, you have to include every supported minor version from your current version up to the target version. These overrides take precedence over the infra operator’s YAOOK_OP_VERSIONS_OVERRIDE. Use at your own risk.

apiVersion: infra.yaook.cloud/v1
kind: AMQPServer
name: "sample-amqp"
spec:
    imageOverrides:
        library/rabbitmq:3.13: custom-url/rabbitmq:v3.13
        library/rabbitmq:4.1: custom-url/rabbitmq:v4.1
        library/rabbitmq:4.2: custom-url/rabbitmq:v4.2
    ...

Using .spec.imageRef to override images is out of support and the setting will be removed in the future.

If the AMQPServer was not created by another operator, you can migrate from imageRef to targetRelease by adding the imageOverrides without removing .spec.imageRef and waiting until the infra operator reconciled the AMQPServer. To upgrade it, remove .spec.imageRef and specify the targetRelease afterwards. Currently, 4.2 is the latest supported release.

Disable quorum queue limits¶

Warning

Disabling the limit is discouraged.

By default, Yaook will limit the size of each quorum queue to 5% of the total available disk space. This ensures that other queues are still usable in case of a broken consumer or a misconfiguration. Disabling the limit increases the timespan until issues occur, but the AMQPServer will become entirely unusable once the limit of the PVC is reached.

To disable the limit, add the following configuration:

apiVersion: infra.yaook.cloud/v1
kind: AMQPServer
name: "sample-amqp"
spec:
    enableQueueSizeLimit: false
    ...

RabbitMQ exporter configuration¶

The RabbitMQ exporter configuration can be modified inside .spec.messageQueue.exporterConfig of the OpenStack k8s manifest, or if you are using a standalone AMQPServer, inside .spec.exporterConfig.

Configuration value	Type	Description
`logLevel="INFO"`	string	Supported values: “DEBUG”, “INFO”, “WARN”, “ERROR”
`accumulatingMessagesThreshold=50`	int	Threshold for the number of messages inside a queue used by the metric `rabbitmq_quorum_queue_accumulating_messages_count`
`accumulatingMessagesCriticalThreshold=500`	int	Threshold for the number of messages inside a queue used by the metric `rabbitmq_quorum_queue_accumulating_messages_critical_count`
`scrapeInterval=60`	int	Scraping interval in seconds. See the RabbitMQ documentation for more information
`httpTimeout=10`	int	Timeout in seconds for requests to the RabbitMQ management API