How to resolve "Leader not available" Kafka error when trying to consume

Question

I'm playing around with Kafka and using my own local single instance of zookeeper + kafka and running into this error that I don't seem to understand how to resolve.

I started a simple server per the Apache Kafka Quickstart Guide

$ bin/zookeeper-server-start.sh config/zookeeper.properties
$ bin/kafka-server-start.sh config/server.properties

Then utilizing kafkacat (installed via Homebrew) I started a Producer that will just echo messages that I type into the console

$ kafkacat -P -b localhost:9092 -t TestTopic -T
test1
test1

But when I try to consume those messages I get an error:

$ kafkacat -C -b localhost:9092 -t TestTopic
% ERROR: Topic TestTopic error: Broker: Leader not available

And similarly when I try to list its' metadata

$ kafkacat -L -b localhost:9092 -t TestTopic
Metadata for TestTopic (from broker -1: localhost:9092/bootstrap):
 0 brokers:
 1 topics:
  topic "TestTopic" with 0 partitions: Broker: Leader not available (try again)

My questions:

1. Is this an issue with my running instance of zookeeper and/or kafkacat - I ask this because I've been constantly shutting them down and restarting them, after deleting the /tmp/zookeeper and /tmp/kafka-logs directories
2. Is there some simple setting that I need to try? I tried adding auto.leader.rebalance.enable=true in Kafka's server.properties settings file, but that didn't fix this particular issue
3. How do I do a fresh restart of zookeeper/kafka. Is shutting them down, deleting the /tmp/zookeeper and /tmp/kafka-logs directories and then restarting zookeeper and then kafka the way to go? (Well maybe the way to go is to build a docker container that I can stand-up and tear down, I was going to use the spotify/docker-kafka container but that is not on Kafka 0.9.0.0 and I haven't taking the time to build my own)

Nebulose

I am having the same issue with the same tutorial. have you found a solution for this? I am not using Kafkakat though. – Dehisce

my solution was to stand up a docker container that I can just tear down and re-run as needed. See: github.com/spotify/docker-kafka you can just change the version of Kafka you want to run in the Dockerfile and build the image – Nebulose

Please have a look here: https://mcmap.net/q/1611775/-apache-kafka-leader_not_available In the situation reported in the other question the problem was seen at production time, but i believe it might be the same error at consumption time as in your case – Dogtired

Can you try bin/kafka-topics.sh --list --bootstrap-server localhost:9092. Try debugging with the built in tools in the bin/ dir. I doubt if your broker is running, you can post your broker logs here to aid in debugging. – Forewoman

Answer 1

1. It might be, but probably is not. My guess is the topic isn't created, so kafkacat echoes the massage on screen but doesn't really send it to kafka. All the topics are probably deleted after you delete the /tmp/kafka-logs
2. No. I don't think this is the way to look for a solution.
3. Having a docker container is definitely the way to go - you'll soon end up running kafka on multiple brokers, examining the replication behavior, high availability scenarios etc.. Having it dockerised helps a lot.

Padegs