No Brokers Available error when trying to connect to Kafka

Question

I have a very strange problem when trying to connect locally to Kafka 0.10.0.0 using Python client on CentOS.

My connection options are pretty simple and default:

kafka_consumer = kafka.KafkaConsumer(
        bootstrap_servers=['localhost:9092'],
        client_id="python-test-consumer"
    )

When I manually set listeners option in Kafka's server.properties file like:

listeners=PLAINTEXT://localhost:9092

I get the kafka.errors.NoBrokersAvailable despite the fact that I can still easily connect to Kafka broker server with curl or other linux stuff.

No advertised.listeners or other deprecated advertised options help to solve the problem. Thus, the only state of configuration which is working is one without listeners. What is certainly unacceptable, because we need to setup local cluster somehow.

It seems that solution for this silly problem is simple and is wondering around, but we couldn't figure it ourselves.

Corruptible

Check out this thread: #35689738 – Princedom

Answer 1

This may sound silly, but the exact same problem happened to me because of this:

I upgraded to Kafka 0.10.0.0 via brew (Mac package manager). Brew then suggests to run like this one-liner:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties; kafka-server-start /usr/local/etc/kafka/server.properties

Instead of how I executed before:

$ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties

The approach suggested kept throwing those "No Brokers Available" errors in the client. Then I just split the command in two lines:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
$ kafka-server-start /usr/local/etc/kafka/server.properties

And everything works like before!

Sorry if this doesn't work for you, but I figured it was worth mentioning.

Pebbly

Do you know any solution that will work for Windows? – Urushiol

Sorry @SteffiKeranRaniJ, I don't know. – Pebbly

OK @Maximiliano – Urushiol

@MaximilianoGuerra After more than a day's struggle, this finally helped me in getting up and running! Thanks a ton. – Woodworker

Answer 2

I was having a similar problem. Turns out the auto version discovery only works with versions of kafka 2.4 or older (see here: https://kafka-python.readthedocs.io/en/master/compatibility.html) and my broker was using a newer version.

By manually setting the kafka version, the problem was fixed for me:

kafka_consumer = kafka.KafkaConsumer(
    bootstrap_servers=['localhost:9092'],
    client_id="python-test-consumer",
    api_version="2.X.X"
)

Monumental