1. Overview
Client applications that use Apache Kafka would usually fall into either of the two categories, namely producers and consumers. Both producers and consumers require that the underlying Kafka server is up and running before they can start their job of producing and consuming, respectively.
In this article, we’ll learn a few strategies to determine if the Kafka server is running.
2. Using Zookeeper Commands
One of the quickest ways to find out if there are active brokers is by using Zookeeper’s dump command. The dump command is one of the 4LW commands available to administer a Zookeeper server.
Let’s go ahead and use the nc command to send the dump command over the Zookeeper server that’s listening at the 2181 port:
$ echo dump | nc localhost 2181 | grep -i broker | xargs
/brokers/ids/0
On executing the command, we see a list of ephemeral broker ids registered with the Zookeeper server. If no ephemeral ids exist, then none of the broker nodes is running.
Further, it’s important to note that the dump command needs to be explicitly allowed in the configuration usually available within either zookeeper.properties or zoo.cfg configuration file:
lw.commands.whitelist=dump
Alternatively, we can also use Zookeeper APIs to find the list of active brokers.
3. Using Apache Kafka’s AdminClient
If our producers or consumers are Java applications, then we can use Apache Kafka’s AdminClient class to find out if the Kafka server is up or not.
Let’s define the KafkaAdminClient class to wrap an instance of the AdminClient class so that we can quickly test our code:
public class KafkaAdminClient {
private final AdminClient client;
public KafkaAdminClient(String bootstrap) {
Properties props = new Properties();
props.put("bootstrap.servers", bootstrap);
props.put("request.timeout.ms", 3000);
props.put("connections.max.idle.ms", 5000);
this.client = AdminClient.create(props);
}
}
Next, let’s define the verifyConnection() method in the KafkaAdminClient class to verify if the client can connect with running broker servers:
public boolean verifyConnection() throws ExecutionException, InterruptedException {
Collection<Node> nodes = this.client.describeCluster()
.nodes()
.get();
return nodes != null && nodes.size() > 0;
}
Finally, let’s put our code to test by connecting to a running Kafka cluster:
@Test
void givenKafkaIsRunning_whenCheckedForConnection_thenConnectionIsVerified() throws Exception {
boolean alive = kafkaAdminClient.verifyConnection();
assertThat(alive).isTrue();
}
4. Using the kcat Utility
We can use the kcat (formerly kafkacat) command to find out if there are running Kafka broker nodes. To do so, let’s use the -L option to show the metadata of an existing topic:
$ kcat -b localhost:9092 -t demo-topic -L
Metadata for demo-topic (from broker -1: localhost:9092/bootstrap):
1 brokers:
broker 0 at 192.168.1.53:9092 (controller)
1 topics:
topic "demo-topic" with 1 partitions:
partition 0, leader 0, replicas: 0, isrs: 0
Next, let’s execute the same command when the broker nodes are down:
$ kcat -b localhost:9092 -t demo-topic -L -m 1
%3|1660579562.937|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)
For this case, we get the “Connection refused” error as there are no running broker nodes. Additionally, we must note that we were able to fail fast by restricting the request timeout to 1 second with the -m option.
We can rely on UI tools such as Offset Explorer for experimental POC projects that don’t require automated checks. However, this approach is not recommended if we want to verify the state of broker nodes for enterprise-grade Kafka clients.
Let’s use the Offset Explorer to connect to the Kafka cluster using the Zookeeper host and port details:
We can see the list of running brokers on the left side pane. That’s it. We got it at a click of a button.
6. Conclusion
In this tutorial, we explored a few command-line approaches using Zookeeper commands, Apache’s AdminClient, and the kcat utility, followed by a UI-based approach to determine whether the Kafka server is up or not.
As always, the complete source code for the tutorial is available over on GitHub.
