RabbitMQ cluster as a single Docker Swarm service

I once configured RabbitMQ cluster as a single Docker Swarm service. Time has come to share this way with the world.

If you later find this article useful take a look at the disclaimer for information on how to thank me.

Introduction

I’ll assume you know what RabbitMQ is and why we need to configure its cluster.

Several years ago, I had to write a script which configures RabbitMQ cluster. RabbitMQ was deployed as a containerized workload orchestrated by Docker Swarm. The script had to handle various use cases: first deployment, redeployment, reboot of VMs, adding one restarted replica to the cluster, adding 2 replicas, etc… It’s important to note that the service was part of PaaS where all services were deployed using deployment script wrapping docker stack cli. We’ll call it deployer.sh. So to deploy RabbitMQ, one had to run ./deployer.sh -deploy rabbitmq.

One of the main challenges was creating a single Docker Swarm service. Interestingly, developers I knew proposed to configure several services like rabbitmq-1, rabbitmq-2, rabbitmq-3. Moreover, some open-source repositories showed how to do it. I even heard that someone did that in the industry in the past. Regardless, it just didn’t make sense to me. We’ll see below why and how I eventually managed to configure it.

First we’ll discuss why to configure RabbitMQ cluster as a single service. Then I’ll show how to configure it.

Why to configure RabbitMQ cluster as a single Docker Swarm service

In short, because you want to treat RabbitMQ service as a single unit. That might be needed while scaling the service, for example. If you have 3 services like rabbitmq-1, rabbitmq-2, rabbitmq-3 how do you scale them? I would call their automatic management messy to say the least.

The same applies to deployment using deployer.sh. How would you deploy, undeploy, redeploy RabbitMQ when you have 3 services? The answer is obvious. However, it’s just a bad way to manage a single service.

Reasoning and implementation should treat RabbitMQ as a single service. It will just have 3 replicas and cluster configured inside the containers. Let’s now see a basic demo on how to do it.

RabbitMQ cluster as a single Docker Swarm service demo

Demo Prerequisites

Install and configure on your machine:

• docker
• docker-swarm
• git

The tutorial assumes familiarity with basic git, docker and docker-compose commands.

Demo

Firstly, clone rabbitmq_cluster repository from my GitHub. Next follow below steps:

• Create docker config:

docker config create rabbitmq-config rabbitmq.conf

• Create swarm network:

docker network create --driver overlay prod-net

• Deploy rabbitmq cluster as docker swarm service:

docker stack deploy -c docker-compose.yml cluster

• Find virtual ip of each container

docker exec -i [container_ip] hostname -i

• Modify /etc/hosts of each container to contain entries of other 2 containers:

For example for cluster_rabbitmq.1... container run:

docker exec -it [rabbitmq1_container_ip] bash bash-5.0# echo "10.0.1.9 rabbitmq-2" >> /etc/hosts bash-5.0# echo "10.0.1.10 rabbitmq-3" >> /etc/hosts

• Verify cluster is configured by running in any container:

docker exec -it [rabbitmq1_container_ip] bash

bash-5.0# rabbitmqctl cluster_status

You should see 3 nodes in cluster:

...
Running Nodes

rabbit@rabbitmq-1
rabbit@rabbitmq-2
rabbit@rabbitmq-3
...

RabbitMQ cluster configuration summary

RabbitMQ cluster is configured using classic_config in rabbitmq.conf.

See details here

cluster_formation.classic_config.nodes.1 = rabbit@rabbitmq-1
cluster_formation.classic_config.nodes.2 = rabbit@rabbitmq-2
cluster_formation.classic_config.nodes.3 = rabbit@rabbitmq-3

• rabbitmq-* became the container name thanks to the service template "rabbitmq-{{.Task.Slot}}". See details about service-templates here.
• rabbit@rabbitmq-* is the expected node name from the point of view of RabbitMQ cluster.

Summary

So, to implement the above idea I automated it using bash script. Today, of course, I’d use Kubernetes instead of Docker Swarm. Moreover, Kubernetes operator would replace the script. Maybe, you did it already and want to share your project? Feel free to do so in the comments.

That’s it about configuring RabbitMQ cluster as a single Docker Swarm service. As always, feel free to share. If you found this article useful, take a look at the disclaimer for information on how to thank me.

Bonus: Recommended RabbitMQ courses on Pluralsight:

RabbitMQ by Example

Scaling Applications with Microservices, MassTransit, and RabbitMQ

Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription)

Recommended books on Amazon:

RabbitMQ in Depth

RabbitMQ Essentials: Build distributed and scalable applications with message queuing using RabbitMQ, 2nd Edition