Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?

Question

I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even do a bin/kafka-topics.sh --describe --zookeeper 172.X.X.X:2181 --topic remotetopic which gives me

Topic:remotetopic PartitionCount:1 ReplicationFactor:3 Configs: Topic: remotetopic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Blockquote

However when i do bin/kafka-console-consumer.sh --zookeeper 172.X.X.X:2181 --from-beginning --topic remotetopic I get

WARN Fetching topic metadata with correlation id 0 for topics [Set(remotetopic)] from broker [id:0,host:localhost,port:9092] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException

Why is the consumer trying to read from localhost? Is there any option or commandline or default file where this is read from; and I can change it?

Any help would be appreciated!

Mide

can you please select the most helpful answer from the given answers? – Parotitis

Answer 1

I had same problem , I am using kafka 0.11 and I started producer and broker in my Ubuntu VM and consumer in Windows 10.

I had changed (and uncommented)

advertised.listeners=PLAINTEXT://your.host.name:9092

to

advertised.listeners=PLAINTEXT://<myActual IP address>:9092

example:

advertised.listeners=PLAINTEXT://192.168.150.150:9092

from file config/server.properties at server/broker side (i.e. in my case it was Ubuntu VM)

Parotitis

An easier way is to use 0.0.0.0 instead of an actual ip address like 192.168.150.150 – Naples

According to the (docs)[kafka.apache.org/documentation/…, it is not valid to set 0.0.0.0 as ip address – Expurgate

Answer 2

Had the same problem. Fixed it by setting the property 'advertised.host.name' as instructed by the comments in the kafka configuration (config/server.properties), and restarting the kafka server.

Ossieossietzky

"advertised.host.name = localhost" replaced 'localhost' with my ip address..thank you – Preemie

Can we then remotely access the logs and get the information on a Java or Python client? – Longerich

Answer 3

I have created a python script which allows you to access kafka/zookeeper via ssh tunnel to consume and produce messages from your local machine.

Compatible for Linux & Mac

Usage: kafkatunnel.py [OPTIONS] COMMAND [ARGS]...

  Access kafka/zookeeper via ssh tunnel to consume and produce messages from
  your local machine

Commands:
  aws     retrieve kafka/zookeeper ip's from AWS...
  manual  provide the IP's of your zookeeper/kafka

try it: https://github.com/simple-machines/kafka-tunnel

Transversal

Answer 4

I have setup Kafka and Zookeeper in a similar (and even more complex) arrangement and the kafka-console-consumer.sh does work as advertised.

This is very likely a mis-configured Kafka instance (which specifies its zookeeper nodes in the server configuration).

Check the Kafka configuration. By default if the zookeeper instances are NOT specified, it presumes a singular localhost instance. Otherwise the "server.1" ... 2 etc convention must be used to define the zookeeper instances (minimum three being recommended/required to obtain proper quorum).

Hellbox