Bootstrap server vs zookeeper in kafka?

Question

Why the use of zookeeper in kafka-consumer is deprecated and why it's recommended to use the bootstrap server instead ? what are the advantages of the bootstrap-server?

Meat

Where did you see that ? – Abner

Duplicate : #41774946 – Abner

Thanks for your quick response. I already consulted this topic but i did not understand the advantage of the bootstrap server.@Abner – Meat

Answer 1

Kafka consumer need to commit the offset to kafka and fetch the offset from kafka. Since kafka moved the offset storage from zookeeper to kafka brokers, a kafka consumer does not need to directly communicate with zookeeper, so the new kafka consumer does not need to config the zookeeper.

However, a kafka consumer always needs to connect to kafka brokers (cluster) to send the request to server, the bootstrap-server is just some brokers of this cluster, and using this, consumer could find all the brokers.

Pyx

No matter new or old, clients should be able to get the broker list via zk. Is there any benefit to confuse users by --zookeeper, --bootstrap-server and --broker-list? – Morganstein

@Morganstein I feel like it is a long-term game going into the direction of eliminating zookeeper. As for now they just doing tiny steps to achieve it and remove dependencies – Ress

Answer 2

• In the older version of Kafka (0.9.0) Kafka use to store data on Kafka server and all offset related information like (current partition offsets) were stored in zookeeper. So for a consumer to run it requires both data and metadata. So for getting metadata, it has to call zookeeper. That's why it is using both zookeeper and Kafka. Ex Old Consumer Code
• In new versions of Kafka (i.e 0.10.0 and above) it stores all topic metadata information(total partitions and their current offsets) in the __consumer_offset topic on the same Kafka server. So now only Kafka broker needs to communicate with zookeeper and consumer gets all data and metadata from Kafka broker itself, so it now no longer need to communicate with zookeeper.

Advantage of the current architecture: it's easier to manage data and metadata when they are at the same place.

Valuate

Answer 3

In the current kafka-consumer tool using the --zookeeper or --bootstrap-server arguments distinguish between using the old and the new consumer. The old consumer needs Zookeeper connection because offset are saved there. The new consumer doesn't need Zookeeper anymore because offsets are saved to __consumer_offset topics on Kafka brokers. Using the old consumer is discouraged today so for new applications it's better using the new implementation.

Boise

What are "the old and the new consumer"? Are you talking about consumers in different versions of Kafka? – Considered

yes the "new" consumer was named in this way startinf from 0.9.0 version where the consumer stores offsets in a kafka topic and not in zookeeper anymore. – Boise