Error in Publishing message to Kafka topic

Question

I am new to Kafka and try to setup enviroment for it . I am trying to run a single node Kafka but I am getting error in that.

Following following steps on mac

1. brew install zookeeper
2. brew install kafka
3. zkServer start
4.  kafka-server-start.sh /usr/local/etc/kafka/server.properties
5.bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
6.bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
This is a message

But I am getting following error. Please let me know , if I missed anything

[2015-10-19 15:48:46,632] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,637] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,638] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-10-19 15:48:46,746] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,749] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,749] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-10-19 15:48:46,860] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,863] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,863] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-10-19 15:48:46,973] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,977] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:46,977] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-10-19 15:48:47,083] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-10-19 15:48:47,084] ERROR Failed to send requests for topics test with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2015-10-19 15:48:47,086] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
    at scala.collection.immutable.Stream.foreach(Stream.scala:547)
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)

thanks

Katharynkathe

Answer 1

Alok, This very common issue of port and host.
Could you please confirm the HOST and PORT for Zookeeper and Kafka Broker. I'm assuming you have configured Kafka in your local so make sure you are providing proper host name.

Run bellow command to make sure the Zookeeper and Kafka Broker running on port which you have mentioned in your command :

netstat -a | grep port

Please let me know if you still face any issue.

Wake

Hi, Port seems correct to me. I verified in zoo.cfg and server.properties. I am using localhost in place of hostname. – Katharynkathe

can you run command "hostname" on your CLI and verify what is the hostname for your machine. – Wake

Answer 2

The problem on my side has been solved by setting the value localhost to the following properties inside the config/server.properties file:

host.name
advertised.host.name

Broad

Thanks, this worked for me. I will also add the need to setting advertised.port=yourPortNum, specially if running on a local machine and the brokers have different ports. – Battalion

these are now deprecated you can just remove them in favour of listeners or advertised.listeners – Blakeslee

Answer 3

Try starting the zookeeper with the proper configuration like you do with the kafka:

bin/zookeeper-server-start.sh config/zookeeper.properties

You may delete the broker log and then try again:

rm -R /tmp/kafka-logs (default location)

If you still have problems check the logs.

Poisson

I am starting with default configuration . I didnt change zookeeper.properties. – Katharynkathe

Answer 4

I solved it by changing the host.name property in server.properties file. By default it is commented out and points to local machine. So, if you want to start kafka locally, just uncomment this out and you are all set. Meaning - host.name=localhost

Louannlouanna

Answer 5

i had set the localhost to 127.0.0.1 to fix this issue in server.properties

Microcosm

Answer 6

While using Kafka broker, zookeeper is must. You must start zookeeper-server before starting kafka-server.

Be very careful with ports, when to use zookeeper port, when to use kafka broker port

If you are getting any exception related to Metadata, Check if your topic exist in broker.

./kafka-topics.sh --zookeeper <host>:<port> --describe --topic <topic-name>

For topics and Consumers use zookeeper port(service) and for consumer use kafka broker.

Berzelius