Kafka throws java.nio.channels.ClosedChannelException

Question

When i try to consume messages from the kafka server which is hosted in ec2 with kafka console tool (V 0.9.0.1 , i think this uses old consumer APIs) I get following exception. How can i overcome this?

#./kafka-console-consumer.sh --zookeeper zookeeper1.xx.com:2181 --topic MY_TOPIC --from-beginning

[2016-04-06 14:34:58,219] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1014,kafka3.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-04-06 14:34:58,222] WARN Fetching topic metadata with correlation id 0 for topics [Set(MY_TOPIC)] from broker [BrokerEndPoint(1013,kafka22.xx.com,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-

Harm

Answer 1

The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file

Harm

Thanks a lot for hint. I had same problem for a while both in my centos and windows 10. As soon as I fixed ´hosts´ file in both machines error disappeared! – Salmonoid

Answer 2

I was able to solve this issue by setting advertised.host.name to the config file

Scopolamine

Answer 3

This is actually a WARNING - no big deal. May be your topic is corrupted? Try to recreate topic.

Darien

I don't think for all topics it gives same type issue pls have a look #36443168 – Harm

I run into several issues with new consumer, it goes infinite loop without polling messages. So, choking with older consumer APIs(the consumer console tools with the kafka distribution) and now stated to get this type closed channel issues – Harm

Here are some issues i face #36394367 #36440318 – Harm

@Harm Ok. If you running kafka on EC2 there may be an issue with private IP's - it can be resolved by setting advertised.host.name=<your external EC2 hostname>, try to search "kafka on EC2 advertised.host.name" or on #30881311 – Darien

The reason for the original closed channel exception is, i had some DNS issue, which i have solved by editing my local hosts file. And the EC2 case, yes i believe my settings are right – Harm

Check this thread at user kafka list //Is there any behavioural change to connect local server and remote server?// – Harm

Answer 4

restating the kafka and service resolved my issue.

Unification