Why can't Kafka Producer connect to zookeeper to fetch broker metadata instead of connecting to brokers

Question

Kafka Producers need a bootstrap list of kafka brokers to work. As per this explanation , it needs this to be able to connect to one of the brokers and then fetch metadata about all live brokers in the cluster.

Now, all the brokers are already registered in the Zookeeper and Kafka consumers connect to ZK which handles from which broker , which partition is data to be read from. Why can't Producers also connect to ZK when ZK already has all the information?

I see there are a few SO questions on this but they seem to explain why consumer needs ZK & not why Producer needs bootstrap list of brokers instead of ZK?

Convertite

Answer 1

Both producers and consumers used to connect to Zookeeper for coordination in the past but has now evolved away from that. With the new consumer API in Kafka 0.9 neither of the clients need to know about Zookeeper anymore and this seems to be the future for Kafka clients. Apart from getting rid of dependencies and flaky connection libraries to ZK, it also makes evolution of the client protocol easier since this is now completely managed by Kafka.

Currently both consumer clients are available in Kafka 0.9 but you should be prepared that the consumer client using ZK connectivity will probably become deprecated at some point in a future release.

Shaft

I am using 8.2.1 version of clients. My Producer needs a bootstrap list of brokers and my consumer needs zookeeper connection. Was wondering why is it kept like this – Convertite

You can kind of see this evolution of it in the documentation pages from kafka.apache.org/07/quickstart.html where both producer and consumer examples are using ZK to the last lines of this section kafka.apache.org/documentation.html#security_ssl shows both producer and consumer connecting to brokers directly in the latest release. – Shaft