Viewing Retention for a Kafka Topic

1. Introduction

In this tutorial, we’ll learn how to get the retention configuration for a particular topic in Kafka. In general, there are a couple of ways to do it. We will explore the approach via the Kafka CLI since it’s the most portable one. Also, we’ll briefly review other approaches.

2. The Default Topic Configuration

Kafka provides us with the kafka-topics.sh script, which allows us to view, alter, create, and delete topics. Let’s create the tasks topic for demonstration purposes:

kafka-topics.sh --bootstrap-server localhost:9092 \
--topic tasks \
--create \
--partitions 1 \
--replication-factor 1

There is also a utility called kafka-configs.sh that comes with most Kafka distributions. This script provides the users with an API to read, alter, delete, or add configuration parameters to our topics. For instance, we can list all of the configs for our topic, including retention configuration:

kafka-configs.sh --bootstrap-server localhost:9092 --describe --topic tasks --all

Make sure to specify an appropriate broker address and topic name — in our case, the topic name is tasks. This way, we can get all configurations defined for this topic, including defaults. To find the retention config only, we can grep for it:

kafka-configs.sh --bootstrap-server localhost:9092 --describe --topic tasks --all | grep retention

This way, we will see only retention configuration for our topic.

3. See Modified Retention Topics

As of Kafka version 3.4.0, the default retention of a Kafka topic is one week. This means each message inside the topic remains for one week. Kafka will purge messages that are older than one week.

Sometimes, when we want to see the retention of Kafka topics, we just want to see the topics with modified retention, not with the default one. Let’s create another topic with a modified retention configuration:

kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic notifications \
--partitions 1 \
--replication-factor 1 \
--config retention.ms=3600000 \
--config retention.bytes=1048576

Here, we created the notifications topic with a retention time of one hour and a retention size of one MB. This means that Kafka removes messages older than one hour. Also, if the topic size exceeds one MB, the oldest records will be removed to maintain one MB as the max size of the topic.

To view the topics with overridden configuration, we can still use kafka-topics.sh with the additional –topics-with-overrides parameter:

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topics-with-overrides

This will produce the following output:

Topic: notifications    TopicId: <TOPIC_ID>    PartitionCount: 1   ReplicationFactor: 1
Configs: retention.ms=3600000,retention.bytes=1048576

Notice that in this list, there is only one topic: notifications. The first topic, tasks, is not displayed because it has the default configuration.

4. Alternative Approaches

Using kafka-topics.sh and kafka-configs.sh, we can get everything that we need. However, it might be valuable to have some kind of UI that can show the same thing. UIs can also help inexperienced bash users view the retention policy. Both Kafka-UI and AKHQ UI are open-source tools for managing Kafka installations. Once we install and configure them, they’ll allow us to view our Kafka topics and settings.

5. Conclusion

In this article, we’ve seen a few ways to check a topic’s retention configuration. We can list all configurations of a particular topic using the kafka-configs.sh utility. Or, if we know that the topic has modified retention configs, we can use kafka-topics.sh to list topics with overridden configuration. There are also several alternative open-source UIs to view the retention settings of a Kafka topic. They will require some extra configuration but still are great choices.