top of page
Programming

Beginners guide to RabbitMQ - (Backend Development Series)

Updated: Dec 9, 2022

Open source multiprotocol Distributed Message Queue


This is the most complete tutorial on rabbitmq with node.js for beginners on the internet.


The best part?

Everything in this tutorial works flawlessly without any errors.


If you are someone who wants to exploreRabbitMQ or message queues in general or are starting to work on RabbitMQ or researching RabbitMQ as a message queue service, this blog is for you. This guide will give you an understanding of what is a rabbitmq message queue, how the publisher-consumer model would work for rabbitmq, what are the steps to set up a rabbitmq connection, and also set up the publisher/consumer connections using nodejs. By the end of this guide, you will be able to create a basic task manager, where the publisher will publish messages and the desired consumer will receive the messages.


RabbitMQ has over 10.1K GitHub stars and 3.7K GitHub forks. Here’s a link to RabbitMQ's open-source repository on GitHub - https://github.com/rabbitmq/rabbitmq-server


Let's dive right into the RabbitMQ tutorial -


Contents


1. What is RabbitMQ? What problem does it solve?

RabbitMQ is an open-source distributed message queue written in Erlang. It supports many communication protocols. In this blog, I will be using the AMQP (Advanced Message Queuing Protocol) - it's a set of standards to be followed while communicating.

Note: RabbitMQ is also called a message broker service or message-oriented middleware.

When to use RabbitMQ?

RabbitMQ was introduced to solve the problem of Spaghetti Mesh Architecture. In this architecture, every client talks to every other client to get the work done. The problem with this is any change made in any one of the clients would impact all the other clients it is connected to, which is obviously not optimal. A middleman/middle layer is required to handle these connections. This is where a message broker like rabbitmq comes in.




2. Pieces of the RabbitMQ Architecture

So how does RabbitMQ work? The RabbitMQ architecture basically comprises 6 pieces. These are


  1. Rabbitmq server - The middleman/middle layer that we talked about above is actually the RabbitMQ server. It always listens to messages ( on port 5672 by default). It has to listen always as it is using TCP.

  2. Publisher - This is basically a client that wants to send some information/message to another client. The publisher establishes a stateful 2 way TCP connection between itself and the RabbitMQ server. The publisher basically tells the rabbitmq server - "Hey, I want to send this message "foobar" to client number 2". It uses the Advanced Message Queue Protocol (AMQP) protocol for this, (AMQP is basically a variant of the TCP protocol).

  3. Consumer - The consumer tells the server - "The line is open for sending messages." the RabbitMQ server would then send messages to the client as and when new messages are received by the server from the publishers. It also establishes a stateful 2 way TCP connection between itself and the RabbitMQ server and it also uses the Advanced Message Queue Protocol (AMQP) protocol. ( Note: If you want to know more about the publisher-subscriber model, check out this article - https://www.thegeekyminds.com/post/publisher-subscriber-model-system-design-architecture )

  4. Channel - A channel is basically a smaller version of the connection. One connection can have multiple channels. The publisher can send messages on specific channels and the consumer can choose to listen on one or multiple channels. The goal here is that not every message the publisher sends needs to be sent to all the consumers. For example - A publisher sends out a message on a channel named "football_score". There can be many consumers connected to the rabbitmq server but the consumers who are listening to the channel named "football_score" will receive the message. It is basically an application of multiplexing. ( Note: Multiplexing is a technique used to combine and send multiple data streams over a single medium. )

  5. Queue - The `Q` in RabbitMQ stands for "Queue". Everything that is received by the RabbitMQ server goes into this queue. All the messages in the server's queue are pushed to the consumers. Dequeue happens when the consumer tells the RabbitMQ server to dequeue the message.

  6. Exchange - Exchange takes care of all the messages received by the rabbitmq server. Whenever a message is received by the server, it goes to the exchange which then takes care of enqueueing/dequeuing the messages. To know more about RabbitMQ exchanges, refer to this link - https://hevodata.com/learn/rabbitmq-exchange-type/




3. Install RabbitMQ and start the RabbitMQ server


To install rabbitmq on your MacBook run these commands

$ brew update
$ brew install rabbitmq
$ export PATH=$PATH:/usr/local/sbin

To install RabbitMQ in Debian and Ubuntu: https://www.rabbitmq.com/install-debian.html

To install RabbitMQ in RPM based Linux: https://www.rabbitmq.com/install-rpm.html

To install RabbitMQ in Windows: https://www.rabbitmq.com/install-windows.html


Rabbitmq is written in Erlang, so you would need to install that dependency first.

After installing, go ahead and start your RabbitMQ server by typing the following command in your terminal:

$ rabbitmq-server


Alternatively, you can also start your rabbitmq server using docker if you have it installed on your machine.


Optional: Running RabbitMQ on a Docker container

(Note: This is an optional step. You can use RabbitMQ without docker. Follow the instructions above for that)


You can use the official docker image for rabbitmq. Run the following command on your terminal to run the rabbitmq server

$ docker run --name rabbitmq -p 5672:5672 -p 15672:15672 --name rabbitmq-server rabbitmq:management

Here, port 5672 is the default port RabbitMQ uses for the Advanced Message Queue Protocol (AMQP) protocol.

Link to the official RabbitMQ Docker image: https://hub.docker.com/_/rabbitmq


Port 15672 is the default port RabbitMQ uses for the HTTP protocol. You can verify that the rabbitmq server has started by going to the following link on your browser. http://localhost:15672/

The default credentials to log in are:

  • username: guest

  • password: guest

If you see a page similar to the following, it means that your RabbitMQ server is up.


Now that we have the RabbitMQ server up and running, let's try to build a Task Manager, where the publisher can send messages to specific channels to the RabbitMQ server and the consumers subscribed to that channel can listen to it.


4. Code a RabbitMQ Publisher Client

I am going to use node.js (nodejs) to code for the publisher. But you can code for it in other languages like Python, Java, C++, etc.


As you know, RabbitMQ uses the Advanced Message Queue Protocol (AMQP) protocol. I will need to install this dependency in my node using npm.

$ npm install amqplib

I will create a file `publisher.js` and start writing

const amqp = require ("amqplib")

because my code requires the Advanced Message Queue Protocol (AMQP) protocol library.


Let's say I want to send a message, "Switch on the light" to a consumer. Let's create a JSON object for this message

const msg = {instruction: "Switch on the light"}

const connection = await amqp.connect("amqp://localhost:5672");

The above line to make a connection to the RabbitMQ server. Using `awake` because "amqp.connect()" will return a Promise.


const channel = await connection.createChannel();

To create a channel on this connection. One connection can have multiple channels.


const assertQ = await channel.assertQueue("instruction");

The function "channel.assertQueue()" will basically assert that the channel named "instruction" actually exists on the server. If it doesn't it will create a channel named "instruction" for you.


Now it's time to send a message to the queue,

await channel.sendToQueue("jobs", Buffer.from(JSON.stringify(msg)))

Finally, close the connection with these lines of code

await channel.close();
await connection.close();


Compiling all of these, my "publisher.js" would look like

const amqp = require ("amqplib")
const msg = {instruction: "Switch on the light"}

async function connect (){
    const connection = await amqp.connect("amqp://localhost:5672");
    const channel = await connection.createChannel();
    const assertQ = await channel.assertQueue("instruction");
    await channel.sendToQueue("jobs", Buffer.from(JSON.stringify(msg)));
    console.log(`Instruction sent successfully ${msg.number}`);
    await channel.close();
    await connection.close();
}

connect();


You can kill the publisher connection after it's done sending the message.


To run the script enter the following command in the terminal window

$ node publisher.js

Now that we have published. How about we consume the message from our consumers?


5. Code a RabbitMQ Consumer Client

The part of the code for connecting to the RabbitMQ server and asserting the channel is the same.

const connection = await amqp.connect("amqp://localhost:5672");
const channel = await connection.createChannel();
const assertQ = await channel.assertQueue("instruction");

My consumer.js looks like this -

const amqp = require ("amqplib")

async function connect (){
    const connection = await amqp.connect("amqp://localhost:5672");
    const channel = await connection.createChannel();
    const result = await channel.assertQueue("instruction");
      
    channel.consume("jobs", message => {
        const content_msg = JSON.parse(message.content.toString());
        console.log(content_msg);
    })
}

connect();

In the publisher, it was okay to close the connection. But in the case of consumers, as it always needs to listen for messages, closing the connection does not make sense.

To run the script enter the following command in the terminal window

$ node consumer.js

Well, there's a problem! Every time I start the consumer, I get the same message back. Even though my consumer has seen the message.

Actually, this is not a problem, this is a safety feature. The reason we keep the same message over and over again is that we did not tell the server that we have received the message. The idea is, the consumer needs to explicitly acknowledge that the message has been received and processed after which the consumer will tell the server that it's safe to remove the message. The RabbitMQ server will then Dequeue the message.


It can be done with this line:

channel.ack(message);  

The above code will tell the server - "Please dequeue the message, I have processed it! "


So my final code would look like this -

const amqp = require ("amqplib")

async function connect (){
    const connection = await amqp.connect("amqp://localhost:5672");
    const channel = await connection.createChannel();
    const result = await channel.assertQueue("instruction");
      
    channel.consume("jobs", message => {
        const content_msg = JSON.parse(message.content.toString());
        console.log(content_msg);
        if (proccess(content_msg) == true){
            channel.ack(message);  
        }
    })
}

connect();