Beginners guide to RabbitMQ - (Backend Development Series)
Updated: Dec 9, 2022
Open source multiprotocol Distributed Message Queue
This is the most complete tutorial on rabbitmq with node.js for beginners on the internet.
The best part?
Everything in this tutorial works flawlessly without any errors.
If you are someone who wants to exploreRabbitMQ or message queues in general or are starting to work on RabbitMQ or researching RabbitMQ as a message queue service, this blog is for you. This guide will give you an understanding of what is a rabbitmq message queue, how the publisher-consumer model would work for rabbitmq, what are the steps to set up a rabbitmq connection, and also set up the publisher/consumer connections using nodejs. By the end of this guide, you will be able to create a basic task manager, where the publisher will publish messages and the desired consumer will receive the messages.
RabbitMQ has over 10.1K GitHub stars and 3.7K GitHub forks. Here’s a link to RabbitMQ's open-source repository on GitHub - https://github.com/rabbitmq/rabbitmq-server
Let's dive right into the RabbitMQ tutorial -
Contents
1. What is RabbitMQ? What problem does it solve?
RabbitMQ is an open-source distributed message queue written in Erlang. It supports many communication protocols. In this blog, I will be using the AMQP (Advanced Message Queuing Protocol) - it's a set of standards to be followed while communicating.
Note: RabbitMQ is also called a message broker service or message-oriented middleware.
When to use RabbitMQ?
RabbitMQ was introduced to solve the problem of Spaghetti Mesh Architecture. In this architecture, every client talks to every other client to get the work done. The problem with this is any change made in any one of the clients would impact all the other clients it is connected to, which is obviously not optimal. A middleman/middle layer is required to handle these connections. This is where a message broker like rabbitmq comes in.
2. Pieces of the RabbitMQ Architecture
So how does RabbitMQ work? The RabbitMQ architecture basically comprises 6 pieces. These are
Rabbitmq server - The middleman/middle layer that we talked about above is actually the RabbitMQ server. It always listens to messages ( on port 5672 by default). It has to listen always as it is using TCP.
Publisher - This is basically a client that wants to send some information/message to another client. The publisher establishes a stateful 2 way TCP connection between itself and the RabbitMQ server. The publisher basically tells the rabbitmq server - "Hey, I want to send this message "foobar" to client number 2". It uses the Advanced Message Queue Protocol (AMQP) protocol for this, (AMQP is basically a variant of the TCP protocol).
Consumer - The consumer tells the server - "The line is open for sending messages." the RabbitMQ server would then send messages to the client as and when new messages are received by the server from the publishers. It also establishes a stateful 2 way TCP connection between itself and the RabbitMQ server and it also uses the Advanced Message Queue Protocol (AMQP) protocol. ( Note: If you want to know more about the publisher-subscriber model, check out this article - https://www.thegeekyminds.com/post/publisher-subscriber-model-system-design-architecture )
Channel - A channel is basically a smaller version of the connection. One connection can have multiple channels. The publisher can send messages on specific channels and the consumer can choose to listen on one or multiple channels. The goal here is that not every message the publisher sends needs to be sent to all the consumers. For example - A publisher sends out a message on a channel named "football_score". There can be many consumers connected to the rabbitmq server but the consumers who are listening to the channel named "football_score" will receive the message. It is basically an application of multiplexing. ( Note: Multiplexing is a technique used to combine and send multiple data streams over a single medium. )
Queue - The `Q` in RabbitMQ stands for "Queue". Everything that is received by the RabbitMQ server goes into this queue. All the messages in the server's queue are pushed to the consumers. Dequeue happens when the consumer tells the RabbitMQ server to dequeue the message.
Exchange - Exchange takes care of all the messages received by the rabbitmq server. Whenever a message is received by the server, it goes to the exchange which then takes care of enqueueing/dequeuing the messages. To know more about RabbitMQ exchanges, refer to this link - https://hevodata.com/learn/rabbitmq-exchange-type/
3. Install RabbitMQ and start the RabbitMQ server
To install rabbitmq on your MacBook run these commands
$ brew update
$ brew install rabbitmq
$ export PATH=$PATH:/usr/local/sbinTo install RabbitMQ in Debian and Ubuntu: https://www.rabbitmq.com/install-debian.html
To install RabbitMQ in RPM based Linux: https://www.rabbitmq.com/install-rpm.html
To install RabbitMQ in Windows: https://www.rabbitmq.com/install-windows.html
Rabbitmq is written in Erlang, so you would need to install that dependency first.
After installing, go ahead and start your RabbitMQ server by typing the following command in your terminal:
$ rabbitmq-server
Alternatively, you can also start your rabbitmq server using docker if you have it installed on your machine.
Optional: Running RabbitMQ on a Docker container
(Note: This is an optional step. You can use RabbitMQ without docker. Follow the instructions above for that)
You can use the official docker image for rabbitmq. Run the following command on your terminal to run the rabbitmq server
$ docker run --name rabbitmq -p 5672:5672 -p 15672:15672 --name rabbitmq-server rabbitmq:management
Here, port 5672 is the default port RabbitMQ uses for the Advanced Message Queue Protocol (AMQP) protocol.
Link to the official RabbitMQ Docker image: https://hub.docker.com/_/rabbitmq
Port 15672 is the default port RabbitMQ uses for the HTTP protocol. You can verify that the rabbitmq server has started by going to the following link on your browser. http://localhost:15672/
The default credentials to log in are:
username: guest
password: guest
If you see a page similar to the following, it means that your RabbitMQ server is up.
Now that we have the RabbitMQ server up and running, let's try to build a Task Manager, where the publisher can send messages to specific channels to the RabbitMQ server and the consumers subscribed to that channel can listen to it.
4. Code a RabbitMQ Publisher Client
I am going to use node.js (nodejs) to code for the publisher. But you can code for it in other languages like Python, Java, C++, etc.
As you know, RabbitMQ uses the Advanced Message Queue Protocol (AMQP) protocol. I will need to install this dependency in my node using npm.
$ npm install amqplibI will create a file `publisher.js` and start writing
const amqp = require ("amqplib")because my code requires the Advanced Message Queue Protocol (AMQP) protocol library.
Let's say I want to send a message, "Switch on the light" to a consumer. Let's create a JSON object for this message
const msg = {instruction: "Switch on the light"}const connection = await amqp.connect("amqp://localhost:5672");The above line to make a connection to the RabbitMQ server. Using `awake` because "amqp.connect()" will return a Promise.
const channel = await connection.createChannel();To create a channel on this connection. One connection can have multiple channels.
const assertQ = await channel.assertQueue("instruction");The function "channel.assertQueue()" will basically assert that the channel named "instruction" actually exists on the server. If it doesn't it will create a channel named "instruction" for you.
Now it's time to send a message to the queue,
await channel.sendToQueue("jobs", Buffer.from(JSON.stringify(msg)))Finally, close the connection with these lines of code
await channel.close();
await connection.close();Compiling all of these, my "publisher.js" would look like
const amqp = require ("amqplib")
const msg = {instruction: "Switch on the light"}
async function connect (){
const connection = await amqp.connect("amqp://localhost:5672");
const channel = await connection.createChannel();
const assertQ = await channel.assertQueue("instruction");
await channel.sendToQueue("jobs", Buffer.from(JSON.stringify(msg)));
console.log(`Instruction sent successfully ${msg.number}`);
await channel.close();
await connection.close();
}
connect();
Link to the GitHub repository: https://github.com/gouravdhar/rabbitmq-task-scheduler
You can kill the publisher connection after it's done sending the message.
To run the script enter the following command in the terminal window
$ node publisher.jsNow that we have published. How about we consume the message from our consumers?
5. Code a RabbitMQ Consumer Client
The part of the code for connecting to the RabbitMQ server and asserting the channel is the same.
const connection = await amqp.connect("amqp://localhost:5672");
const channel = await connection.createChannel();
const assertQ = await channel.assertQueue("instruction");
My consumer.js looks like this -
const amqp = require ("amqplib")
async function connect (){
const connection = await amqp.connect("amqp://localhost:5672");
const channel = await connection.createChannel();
const result = await channel.assertQueue("instruction");
channel.consume("jobs", message => {
const content_msg = JSON.parse(message.content.toString());
console.log(content_msg);
})
}
connect();
In the publisher, it was okay to close the connection. But in the case of consumers, as it always needs to listen for messages, closing the connection does not make sense.
To run the script enter the following command in the terminal window
$ node consumer.js
Well, there's a problem! Every time I start the consumer, I get the same message back. Even though my consumer has seen the message.
Actually, this is not a problem, this is a safety feature. The reason we keep the same message over and over again is that we did not tell the server that we have received the message. The idea is, the consumer needs to explicitly acknowledge that the message has been received and processed after which the consumer will tell the server that it's safe to remove the message. The RabbitMQ server will then Dequeue the message.
It can be done with this line:
channel.ack(message); The above code will tell the server - "Please dequeue the message, I have processed it! "
So my final code would look like this -
const amqp = require ("amqplib")
async function connect (){
const connection = await amqp.connect("amqp://localhost:5672");
const channel = await connection.createChannel();
const result = await channel.assertQueue("instruction");
channel.consume("jobs", message => {
const content_msg = JSON.parse(message.content.toString());
console.log(content_msg);
if (proccess(content_msg) == true){
channel.ack(message);
}
})
}
connect();
Link to the GitHub repository: https://github.com/gouravdhar/rabbitmq-task-scheduler
6. Creating multiple consumers listening on various Channels
For this example, I am using 2 channels - "instruction", and "match_score". Very opposite in terms of functionality whatsoever.
I am using 1 publisher and 3 consumers.
Consumer 1 listening to channel - "instruction"
Consumer 2 listening to channel - "match_score"
Consumer 3 listening to channels - "instruction", and "match_score"
This is clearly illustrated in the diagram above.
Here's the code for publisher.js
consumer1.js
consumer2.js
consumer3.js
Link to the corresponding GitHub repo : https://github.com/gouravdhar/rabbitmq-task-scheduler
Let's look at the outputs
Publisher
Consumer 1
Consumer 2
Consumer 3
7. RabbitMQ UI Management Portal
RabbitMQ provides a very nice user-friendly interface to monitor the server from the web browser. Queues, exchanges, channels, users, and permissions can be created, deleted, and managed from the browser. You can also manage messages and send messages manually. The management portal can be found on the default port 15672 of the HTTP protocol. For local, the URL would be http://localhost:15672. The default username and password for logging in are "guest" and "guest" respectively but they can be changed after logging in.
The above page should be visible after logging in. This page has 5 tabs - Overview, Connections, Channels, Exchanges, Queues, Admin
Overview
The above image has the overview tab opened. It shows the queued messages and the message rates. "Ready" shows the number of messages that are actually ready to be delivered. "Unacked" shows the number of messages that have been delivered and the server is waiting for them to be acknowledged. If you see that the number of unacked messages is going high, it is an indication that your consumers are not able to process all the messages and maybe it's time to scale up your system.
The nodes and ports information can be seen in the respective section.
Connections
This tab shows the connections established with the RabbitMQ server. It gives information on the user connected to the cluster and the number of channels it is using. The SSL/TLS column tells whether the connection is secured. If you click on a connection, it will give a summary of that connection.
Channels
The channels tab shows the active channels and the user connected to that channel. If you click on a channel you get more information and analytics on that channel.
Exchanges
When a publisher sends a message, it is actually received by the exchange which then enqueues the message. Features show the parameters for the exchange (e.g. D stands for durable, and AD for auto-delete).
Queues
This shows all the queues that have been created so far. You can also manually create a queue from the interface. Queues have different parameters and arguments depending on how they were created. The features column shows the parameters that belong to the queue. It could be features like
Durable queue - It ensures that RabbitMQ will never lose the queue
Message TTL - tells how long a message published to a queue can live before it is discarded
Auto expire - sets how long a queue can be unused before it is automatically deleted
Max length - specifies how many (ready) messages a queue can contain before it starts to drop them)
Max length bytes - specifies the total body size for ready messages a queue can contain before it starts to drop them
You can also click on a queue to see more details.
Using the publish message button, you can manually add messages to the queue to test the consumers.
The "Get Messages" option can be used to manually see the messages in the queue
Admin
In the admin tab, it is possible to add users and change user permissions. Basically, everything related to users and privileges can be done here.
8. These top companies are using RabbitMQ
The top companies that use RabbitMQ are:
Reddit
Robinhood
Accenture
CircleCI
Alibaba travels
TechStack
Sentry
Trivago
9. My thoughts and Conclusion
RabbitMQ is an easy-to-configure messaging broker. I like the UI and the management portal where you can monitor and debug a lot of things. So, it's a plus one on ease of use and setup. It supports a variety of protocols messaging protocols, so more options to choose from. It also provides integration options with some of the popular tools like Datadog, Buddy, Netdata, TimescaleDB, and StackStorm.
And that's a wrap! Thanks for reading till the end.
Let me know of any other messaging queue that you have used in the comments below.
Subscribe to my newsletter to learn something new every week. I write articles on backend development and System Design. Here's the link - https://thegeekyminds.com/subscribe
10. Frequently Asked Questions (FAQs)
Can RabbitMQ lose messages?
In order to avoid losing messages on the RabbitMQ side, queues and messages must be able to cope with RabbitMQ node restarts, and node and hardware failures. You can learn more in their official Reliability Guide.
Are RabbitMQ messages encrypted?
Where does RabbitMQ store data?
Who creates queues in RabbitMQ?
Can RabbitMQ have multiple queues?
How long do messages stay in RabbitMQ queues?
And that's a wrap! Hi, I am Gourav Dhar, a software developer and I write blogs on Backend Development and System Design. Subscribe to my Newsletter and learn something new every week - https://thegeekyminds.com/subscribe
