Sending Key/Value Messages From the Command Line in Kafka

1. Overview

In this tutorial, we’ll learn two methods for sending key/value messages from the command line in Kafka.

Ensuring the ordering of messages on specific topics is a common requirement in real-life event-driven systems dealing with financial transactions, bookings, online shopping, etc. In these scenarios, we should employ Kafka message keys for the events sent to these topics.

2. Prerequisites

Before sending key/value messages from the command line, we need to check a few things.

First, we need a running Kafka instance. If none is available, we can set up a working environment using the Kafka Docker or Kafka quickstart guides. We’ll proceed to the following sections with the assumption that we have a working Kafka environment accessible at kafka-server:9092.

Next, let’s consider that the messages we send from the command line are part of a payment system. Here’s the corresponding model class:

public class PaymentEvent {
    private String reference;
    private BigDecimal amount;
    private Currency currency;

    // standard getters and setters
}

Another prerequisite is having access to the Kafka CLI tools, which is a straightforward process. We have to download a Kafka release, extract the downloaded file, and navigate to the extracted folder. The Kafka CLI tools are now available in the bin folder. We’ll consider that all the CLI commands in the following sections are executed in the extracted Kafka folder location.

Next, let’s create the payments topic where we’ll send the messages:

bin/kafka-topics.sh --create --topic payments --bootstrap-server kafka-server:9092

We should see the following message in the console, indicating that the topic was successfully created:

Created topic payments.

Finally, let’s also create a Kafka consumer on the payments topic to test that the messages are sent correctly:

bin/kafka-console-consumer.sh --topic payments --bootstrap-server kafka-server:9092 --property "print.key=true" --property "key.separator=="

Note the print.key property at the end of the previous command. The consumer doesn’t print the message key without explicitly setting the property to true. We’re also overriding the default value (\t tab character) of the key.separator property to keep things consistent with the way well produce messages in the following sections.

We’re now ready to start sending key/value messages from the command line.

3. Sending Key/Value Messages From the Command Line

We send the key/value messages from the command line using the Kafka console producer:

bin/kafka-console-producer.sh --topic payments --bootstrap-server kafka-server:9092 --property "parse.key=true" --property "key.separator=="

The parse.key and key.separator properties provided at the end of the previous command are required when we want to provide the message key along with the message payload from CLI.

After running the previous command, a prompt appears where we can provide the message key and message payload:

>KEY1={"reference":"P000000001", "amount": "37.75", "currency":"EUR"}
>KEY2={"reference":"P000000002", "amount": "2", "currency":"EUR"}

We can see from the consumer output that both the message key and message payload are correctly sent from the command line:

KEY1={"reference":"P000000001", "amount": "37.75", "currency":"EUR"}
KEY2={"reference":"P000000002", "amount": "2", "currency":"EUR"}

4. Sending Key/Value Messages From a File

Another approach for sending key/value messages from the command line is to use a file. Let’s see how this works.

First, let’s create the payment-events.txt file with the content below:

KEY3={"reference":"P000000003", "amount": "80", "currency":"SEK"}
KEY4={"reference":"P000000004", "amount": "77.8", "currency":"GBP"}

Now, let’s start the console producer and use the payment-events.txt file as input:

bin/kafka-console-producer.sh --topic payments --bootstrap-server kafka-server:9092 --property "parse.key=true" --property "key.separator==" < payment-events.txt

Looking at the consumer output we can see that both the message key and message payload are correctly sent this time too:

KEY3={"reference":"P000000003", "amount": "80", "currency":"SEK"}
KEY4={"reference":"P000000004", "amount": "77.8", "currency":"GBP"}

5. Conclusion

In this article, we learned how to send key/value messages from the command line in Kafka. We also saw an alternative method for sending a batch of events using an existing file. These methods prove useful when we want to ensure messages are delivered on a specific topic while maintaining the order of messages.