Kafka consumer not able to consume messages using bootstrap server name

Question

I am facing an issue while consuming message using the bootstrap-server i.e. Kafka server. Any idea why is it not able to consume messages without zookeeper?

• Kafka Version: kafka_2.11-1.0.0
• Zookeeper Version: kafka_2.11-1.0.0
• Zookeeper Host and port: zkp02.mp.com:2181
• Kafka Host and port: kfk03.mp.com:9092

Producing some message:

[kfk03.mp.com ~]$ /bnsf/kafka/bin/kafka-console-producer.sh --broker-list kfk03.mp.com:9092 --topic test
>hi
>hi

Consumer not able to consume messages if I give –-bootstrap-server:

[kfk03.mp.com ~]$
/bnsf/kafka/bin/kafka-console-consumer.sh --bootstrap-server kfk03.mp.com:9092 --topic test --from-beginning

Consumer able to consume messages when --zookeeper server is given instead of --bootstrap-server -:

[kfk03.mp.com ~]$ /bnsf/kafka/bin/kafka-console-consumer.sh --zookeeper zkp02.mp.com:2181 --topic test --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
hi
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
{"properties": {"messageType": "test", "sentDateTime": "2018-02-25T21:46:00.000+0000"}, "name": "Uttam Anand", "age": 29}
hi
hi
uttam
hi
hi
hi
hello
hi
^CProcessed a total of 17 messages

Edp

server.properties broker.id=0 listeners=PLAINTEXT://:9092 port=9092 host.name=kfk03.mp.com advertised.host.name=kfk03.mp.com advertised.port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/bnsf/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=1440 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=zkp02.mp.com:2181 zookeeper.connection.timeout.ms=60000 – Edp

Newer versions of Kafka took some processes away from zookeeper and into the Kafka brokers. This enabled them to protect zookeeper from unexpected outages. Although main operational functionalities retained in zookeeper. That's why your consumers are able to consume messages when --zookeeper is given not --bootstrap-server – Spitter

Answer 1

While consuming messages from kafka using bootstrap-server parameter, the connection happens via the kafka server instead of zookeeper. Kafka broker stores offset details in __consumer_offsets topic.

Check if __consumer_offsets is present in your topics list. If it's not there, check kafka logs to find the reason.

We faced a similar issue. In our case the __consumer_offsets was not created because of the following error:

ERROR [KafkaApi-1001] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor').

Goodoh

Tnx for this hint. this costed me 3 h of debugging. cheers again – Snap

Same for me - waster 1 hr - finally found this! – Dalury