In this quick tutorial, we'll learn how we can list all the topics in an Apache Kafka cluster.
First, we'll set up a single-node Apache Kafka and Zookeeper cluster. Then we'll ask that cluster about its topics.
2. Setting Up Kafka
Before listing all the topics in a Kafka cluster, let's set up a test single-node Kafka cluster in three steps:
- Downloading Kafka and Zookeeper
- Starting Zookeeper Service
- Starting Kafka Service
First, we should make sure to download the right Kafka version from the Apache site. Once the download finishes, we should extract the downloaded archive:
$ tar xvf kafka_2.13-2.6.0.tgz
Kafka is using Apache Zookeeper to manage its cluster metadata, so we need a running Zookeeper cluster.
For test purposes, we can run a single-node Zookeeper instance using the zookeeper-server-start.sh script in the bin directory:
$ cd kafka_2.13-2.6.0 # extracted directory $ ./bin/zookeeper-server-start.sh config/zookeeper.properties
This will start a Zookeeper service listening on port 2181. After that, we can use another script to run the Kafka server:
$ ./bin/kafka-server-start.sh config/server.properties
After awhile, a Kafka broker will start. Let's add a few topics to this simple cluster:
$ bin/kafka-topics.sh --create --topic users.registrations --replication-factor 1 \ --partitions 2 --zookeeper localhost:2181 $ bin/kafka-topics.sh --create --topic users.verfications --replication-factor 1 \ --partitions 2 --zookeeper localhost:2181
Now that everything is ready, let's see how we can list Kafka topics.
3. Listing Topics
To list all the Kafka topics in a cluster, we can use the bin/kafka-topics.sh shell script bundled in the downloaded Kafka distribution. All we have to do is to pass the –list option, along with the information about the cluster. For instance, we can pass the Zookeeper service address:
$ bin/kafka-topics.sh --list --zookeeper localhost:2181 users.registrations users.verfications
As shown above, the –list option tells the kafka-topics.sh shell script to list all the topics. In this case, we have two topics to store user-related events. If there's no topic in the cluster, then the command will return silently without any result.
Also, in order to talk to the Kafka cluster, we need to pass the Zookeeper service URL using the –zookeeper option.
It's even possible to pass the Kafka cluster address directly using the –bootstrap-server option:
$ ./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --list users.registrations users.verfications
Our single-instance Kafka cluster listens to the 9092 port, so we specified “localhost:9092” as the bootstrap server. Put simply, bootstrap servers are Kafka brokers.
If we don't pass the information necessary to talk to a Kafka cluster, the kafka-topics.sh shell script will complain with an error:
$ ./bin/kafka-topics.sh --list Exception in thread "main" java.lang.IllegalArgumentException: Only one of --bootstrap-server or --zookeeper must be specified at kafka.admin.TopicCommand$TopicCommandOptions.checkArgs(TopicCommand.scala:721) at kafka.admin.TopicCommand$.main(TopicCommand.scala:52) at kafka.admin.TopicCommand.main(TopicCommand.scala)
As shown above, the shell scripts require us to pass either the –bootstrap-server or –zookeeper option.
4. Topic Details
Once we've found a list of topics, we can take a peek at the details of one specific topic. To do that, we can use the “–describe –topic <topic name>” combination of options:
$ ./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --describe --topic users.registrations Topic: users.registrations PartitionCount: 2 ReplicationFactor: 1 Configs: segment.bytes=1073741824 Topic: users.registrations Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: users.registrations Partition: 1 Leader: 0 Replicas: 0 Isr: 0
These details include information about the specified topic, such as the number of partitions and replicas, among others. Similar to other commands, we must pass the cluster information or Zookeeper address. Otherwise, we won't be able to talk to the cluster.
In this brief article, we learned how to list all the topics in a Kafka cluster. Along the way, we saw how to set up a simple, single-node Kafka cluster.
Currently, Apache Kafka uses Zookeeper to manage its cluster metadata. This, however, will change shortly as part of KIP-500, as Kafka is going to have its own metadata quorum.