1. Overview
In this tutorial, we’ll look at the use of message queues and publishers/subscribers. These are common patterns used in distributed systems for two or more services to communicate with one another.
For this tutorial, all examples will be shown using the RabbitMQ message broker so first follow RabbitMQ’s tutorial to get up and running locally. For a deeper dive on RabbitMQ check out our other tutorial.
Note: there are many alternatives to RabbitMQ that can be used for the same examples in this tutorial such as Kafka, Google Cloud Pub-Sub, and Amazon SQS to name but a few.
2. What Are Message Queues?
Let’s start by looking at message queues. Message queues consist of a publishing service and multiple consumer services that communicate via a queue. This communication is typically one way where the publisher will issue commands to the consumers. The publishing service will typically put a message on a queue or exchange and a single consumer service will consume this message and perform an action based on this.
Consider the following exchange:
From this, we can see a Publisher service that is putting a message ‘m n+1’ onto the queue. In addition, we can also see multiple messages already in existence on the queue waiting to be consumed. On the right-hand side, we have 2 consuming services ‘A’ and ‘B’ that is listening to the queue for messages.
Let’s now consider the same exchange after some time:
First, we can see that the Publisher’s message has been pushed to the tail of the queue. Next, the important part to consider is the right-hand side of the image. We can see that consumer ‘A’ has read the message ‘m 1’ and, as such, it is no longer available in the queue for the other service ‘B’ to consume.
2.1. Where to Use Message Queues
Message queues are often used where we want to delegate work from a service. In doing so, we want to ensure that the work is only executed one time.
Using message queues is popular in micro-service architectures and while developing cloud-based or serverless applications as it allows us to horizontally scale our app based on load.
For example, if there are many messages on the queue waiting to be processed, we can spin up multiple consumer services which listen to the same message queue and handle the influx in messages. Once the messages have been handled, the services can then be turned off when traffic is minimal to save on running costs.
2.2. Example Using RabbitMQ
Let’s go through an example for clarity. Our example will take the form of a pizza restaurant. Imagine people are able to order pizzas via an app and chefs at the pizzeria will pick up orders as they come in. In this example, the customer is our publisher and the chef(s) are our consumers.
First, let’s define our queue:
private static final String MESSAGE_QUEUE = "pizza-message-queue";
@Bean
public Queue queue() {
return new Queue(MESSAGE_QUEUE);
}
Using Spring AMQP, we have created a queue named “pizza-message-queue”. Next, let’s define our publisher that will post messages to our newly defined queue:
public class Publisher {
private RabbitTemplate rabbitTemplate;
private String queue;
public Publisher(RabbitTemplate rabbitTemplate, String queue) {
this.rabbitTemplate = rabbitTemplate;
this.queue = queue;
}
@PostConstruct
public void postMessages() {
rabbitTemplate.convertAndSend(queue, "1 Pepperoni");
rabbitTemplate.convertAndSend(queue, "3 Margarita");
rabbitTemplate.convertAndSend(queue, "1 Ham and Pineapple (yuck)");
}
}
Spring AMQP will create a RabbitTemplate bean for us that has a connection to our RabbitMQ exchange to reduce configuration overhead. Our Publisher makes use of this by sending 3 messages to our queue.
Now that our pizza orders are in we need a separate consumer application. This will act as our chef in the example and read messages:
public class Consumer {
public void receiveOrder(String message) {
System.out.printf("Order received: %s%n", message);
}
}
Let’s now create a MessageListenerAdapter for our queue that will call our Consumer’s receive order method using reflection:
@Bean
public SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(MESSAGE_QUEUE);
container.setMessageListener(listenerAdapter);
return container;
}
@Bean
public MessageListenerAdapter listenerAdapter(Consumer consumer) {
return new MessageListenerAdapter(consumer, "receiveOrder");
}
Messages read from the queue will now be routed to the receiveOrder method of the Consumer class. To run this application, we can create as many Consumer applications as we wish to fulfill the incoming orders. For example, if 400 pizza orders were put on the queue then we may need more than 1 consumer ‘chef’, or orders will be slow. In this case, we might spin up 10 consumer instances to fulfill the orders in a timely manner.
3. What Is the Pub-Sub?
Now that we have covered message queues, let’s look into pub-sub. Conversely, to message queues, in a pub-sub architecture we want all our consuming (subscribing) applications to get at least 1 copy of the message that our publisher posts to an exchange.
Consider the following exchange:
On the left we have a publisher sending a message “m n+1” to a Topic. This Topic will broadcast this message to its subscriptions. These subscriptions are bound to queues. Each queue has a listening subscriber service awaiting messages.
Let’s now consider the same exchange after some time has passed:
Both the subscribing services are consuming “m 1” as both received a copy of this message. In addition, the Topic is distributing the new message “m n+1” to all of its subscribers.
Pub sub should be used where we need a guarantee that each subscriber gets a copy of the message.
3.1. Example Using RabbitMQ
Imagine we have a clothing website. This website is able to send push notifications to users to notify them of deals. Our system can send notifications via email or text alerts. In this scenario, the website is our publisher and the text and email alerting services are our subscribers.
First, let’s define our topic exchange and bind 2 queues to it:
private static final String PUB_SUB_TOPIC = "notification-topic";
private static final String PUB_SUB_EMAIL_QUEUE = "email-queue";
private static final String PUB_SUB_TEXT_QUEUE = "text-queue";
@Bean
public Queue emailQueue() {
return new Queue(PUB_SUB_EMAIL_QUEUE);
}
@Bean
public Queue textQueue() {
return new Queue(PUB_SUB_TEXT_QUEUE);
}
@Bean
public TopicExchange exchange() {
return new TopicExchange(PUB_SUB_TOPIC);
}
@Bean
public Binding emailBinding(Queue emailQueue, TopicExchange exchange) {
return BindingBuilder.bind(emailQueue).to(exchange).with("notification");
}
@Bean
public Binding textBinding(Queue textQueue, TopicExchange exchange) {
return BindingBuilder.bind(textQueue).to(exchange).with("notification");
}
We have now bound 2 queues using the routing key “notification” meaning any messages posted on the topic with this routing key will go to both queues. Updating the Publisher class that we created earlier, we can send some messages to our exchange:
rabbitTemplate.convertAndSend(topic, "notification", "New Deal on T-Shirts: 95% off!");
rabbitTemplate.convertAndSend(topic, "notification", "2 for 1 on all Jeans!");
4. Comparison
Now that we’ve touched on both areas, let’s briefly compare both types of exchange.
As previously mentioned, both message queues and pub-sub architecture patterns are a great way to break up an application to make it more horizontally scalable.
Another benefit of using either pub-sub or message queues is that the communication is more durable than traditional synchronous modes of communication. For example, if app A communicates to app B via asynchronous HTTP call then if either of the applications goes down the data is lost and the request must be retried.
Using message queues if a consumer application instance goes down then another consumer will be able to handle the message instead. Using pub-sub, if a subscriber is down then once it has recovered the messages it has missed will be available for consumption in its subscribing queue.
Finally, context is key. Choosing whether to use pub-sub or message queue architecture comes down to defining exactly how you want the consuming service to behave. The most important factor to keep in mind is asking “Does it matter if every consumer gets every message?”
5. Conclusion
In this tutorial we’ve looked at pub-sub and message queues and some of the characteristics of each. All the code mentioned in this tutorial can be found over on GitHub.
