NoBrokersAvailable: NoBrokersAvailable-Kafka Error

Question

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'.

My kafka is running but when i want to create a partition.

 from kafka import TopicPartition
(ERROR THERE) consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
 consumer.assign([TopicPartition('foobar', 2)])
 msg = next(consumer)

traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/group.py", line 284, in init self._client = KafkaClient(metrics=self._metrics, **self.config) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 202, in init self.config['api_version'] = self.check_version(timeout=check_timeout) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 791, in check_version raise Errors.NoBrokersAvailable() kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Aleedis

Steps to create a Kafka Data pipeline. Follow the link below. #35689738 – Stuffy

Answer 1

You cannot create partitions within a consumer. Partitions are created when you create a topic. For example, using command line tool:

bin/kafka-topics.sh \
  --zookeeper localhost:2181 \
  --create --topic myNewTopic \
  --partitions 10 \
  --replication-factor 3

This creates a new topic "myNewTopic" with 10 partitions (numbered from 0 to 9) and replication factor 3. (see http://docs.confluent.io/3.0.0/kafka/post-deployment.html#admin-operations and https://kafka.apache.org/documentation.html#quickstart_createtopic)

Within your consumer, if you call assign(), it means you want to consume the corresponding partition and this partition must exist already.

Stuff

Answer 2

I had the same error during kafka streaming. The code below resolved my error: We need to define the API version in KafkaProducer.

KafkaProducer(bootstrap_servers=['localhost:9092'],
              api_version=(0,11,5),
              value_serializer=lambda x: dumps(x).encode('utf-8'))

Veloz

could you detail more on how to get that api_version? is that the same as kafka broker version? – Recalesce

please who else know how to get the api_version to be use in the producer class as mentioned by @Veloz – Heat

Answer 3

The problem for me, was the firewall rule as I am running Kafka on Google Cloud.

It was working for me yesterday and I was scratching my head today for 1 hour thinking about why it doesn't work anymore .

As the public IP address of my local system changes every time I connect to a different LAN or WiFi, I had to allow my local system's public IP in the firewall rules. I would suggest using a connection with a fixed public IP or to check for this whenever you switch/change your connection.

These small changes in the configurations take too much to debug and fix them. Felt like wasted an hour for this.

Cheltenham

Answer 4

Don't know if this answer is still relevant but recently resolved this same problem in a VBox VM broker not reachable from host Windows OS. Since you have mentioned bootsrap_servers in KafkaConsumer, I assume you are using at least kafka 0.10.0.0

Please look for the advertised.listeners property in server.properties file and set it to PLAINTEXT://localhost:9092 or PLAINTEXT://<broker_ip>:9092

But before you set that make sure your broker is reachable from the environment where your consumer is running (by doing ping <broker_ip> and netcat nc -vz <broker_ip> 9092).

Also, you need to restart the kafka-server and consumer/producer (whatever is running) and try sending/receiving.

For example, if you are running VM, you may like to use Host-only adapter to make the broker reachable from host machine

NOTE: This configuration works for Kafka Server >= 0.10.X.X but not for 0.8.2.X. Haven't checked for 0.9.0.X

Ritual

Answer 5

It looks like you want to start consuming messages instead of creating partions. Nevertheless - can you reach kafka at port 1234? 9092 is kafkas default port maybe you can try this one. If you found the right port but your application still produces errors you can try to use a console consumer to test your setup:

bin/kafka-console-producer.sh --broker-list localhost:<yourportnumber> --topic foobar

The console consumer is part of the standard kafka distribution. Maybe that gets you a little closer to the source of the problem.

Goddamn

Answer 6

NoBrokersAvailable can be a answer of bad configuration of hostname in kafka configuration.

Arioso

Answer 7

Use 127.0.0.1 instead of localhost or any other IP relevant to your usecase. Changing localhost:9092 to 127.0.0.1:9092 worked for me.

from kafka import KafkaConsumer
consumer = KafkaConsumer('topicname',bootstrap_servers=['127.0.0.1:9092'])
print(consumer.config)
print(consumer.bootstrap_connected())

Bronchopneumonia