What happens if Zookeeper fails completely?

Question

we have setup a Kafka/Zookeeper Cluster consisting of 3 Brokers. We have one producer, sending messages to one specific Kafka topic and a few consumer groups reading from said topic. Those consumers perform a leader election via Zookeeper for themselves (independent from Kafka).

The versions used are:

• Kafka: 0.9.0.1
• Zookeeper: 3.4.6 (included in the Kafka-Package)

All processes are managed by Supervisor. So far, everything works just fine. What we tried now (for testing purposes) was to simply kill off all Zookeeper processes and see what happens.

As we expected, our consumer processes couldn't connect to Zookeeper anymore. But unexpectedly, the Kafka Brokers still worked. Our producer didn't complain at all and was still able to write into the topic. While I couldn't use kafka/bin/kafka-topics.sh or similar, since they all require a zookeeper-parameter, I could still see the actual size of the topic-log grow. After restarting the zookeeper processes, everything again worked just like before.

What we couldn't figure out is now... what actually happened there? We thought, Kafka would require a working Zookeeper-Connection and we couldn't find any explanation for this behaviour online.

Tag

Zookeeper is nor required for all operation done by Kafka. For examples, Kakfa Consumer Clients commit their offsets to ZK. As fas as I know. ZK is also used if a broker fails to elect new leaders for the partition the failed broker hosted. However, as long as all broker are up an working, writing is not problem: See kafka.apache.org/090/documentation.html#replication and confluent.io/blog/… for mere details. – Smoot

> Kakfa Consumer Clients commit their offsets to ZK. Do they? As far as I understood, the "New Consumer" doesn't need that., since they wanted to decouple consumers from Zookeeper. That's why you use the property bootstrap.servers instead of zookeeper.connect and use the Kafka-Ports – Tag

Yes. Old consumer commit offset into ZK. The new consumers commit their offsets into a Kafka topic and are independent of ZK. – Smoot

As per Wiki, Zookeeper is used for: Server failure detection, Data partitioning and In-sync data replication. Pls refer cwiki.apache.org/confluence/display/KAFKA/…? – Weslee

quora.com/… – Weslee

Answer 1

When you have one node of zookeeper, broker will not be able to contact zookeeper, after broker discovers zookeeper is not reachable, broker also will become unreachable. Hence the producer and consumer. In case of producer it starts dropping(reject the record). In case of consumer it can happen that, the read record which is not ack'ed may end up processing again when broker is up and ready...

in case of 3node zk one node failure is acceptable as quorum is still satisfied... but cant afford the 2node failures which will lead to the above consequences...

Blistery