Why do we need ZooKeeper in the Hadoop stack?

Question

I am new to Hadoop/ZooKeeper. I cannot understand the purpose of using ZooKeeper with Hadoop, is ZooKeeper writing data in Hadoop? If not, then why we do we use ZooKeeper with Hadoop?

Lindberg

And where is Zookeeper used in Hadoop? – Monody

Answer 1

Hadoop 1.x does not use Zookeeper. HBase does use zookeeper even in Hadoop 1.x installations.

Hadoop adopted Zookeeper as well starting with version 2.0.

The purpose of Zookeeper is cluster management. This fits with the general philosophy of *nix of using smaller specialized components - so components of Hadoop that want clustering capabilities rely on Zookeeper for that rather than develop their own.

Zookeeper is a distributed storage that provides the following guarantees (copied from Zookeeper overview page):

• Sequential Consistency - Updates from a client will be applied in the order that they were sent.
• Atomicity - Updates either succeed or fail. No partial results.
• Single System Image - A client will see the same view of the service regardless of the server that it connects to.
• Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.
• Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.

You can use these to implement different "recipes" that are required for cluster management like locks, leader election etc.

If you're going to use ZooKeeper yourself, I recommend you take a look at Curator from Netflix which makes it easier to use (e.g. they implement a few recipes out of the box)

Flyboat

When you say ' Hadoop adopted Zookeeper as well starting with version 2.0.', does it mean zookeeper is included in hadoop distribution ver 2.0 onwards? – Subacid

Since most distribution included HBase it was there before v.2. In V2. YARN also uses zookeeper for HA (actually you can use less reliable ways but it is the recommended way see for example blog.cloudera.com/blog/2014/05/how-apache-hadoop-yarn-ha-works) so I don't think you'd find or create a distribution without it – Flyboat

Answer 2

Zookeeper solves the problem of reliable distributed coordination, and hadoop is a distributed system, right?

There's an excellent paper Paxos Algorithm that you can read on this subject.

Deceased

stack overflow 101: in most cases the shorter the better – Hertzog

For anyone finding Paxos difficult to understand, Raft is an easier-to-understand equivalent. – Rains

Answer 3

From zookeeper documentation page:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

From hadoop documentation page:

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models

Regarding your query:

Why do we need ZooKeeper in Hadoop Stack?

The binding factor is distributed processing and high availability.

e.g. Hadoop Namenode fail over process.

Hadoop high availability is designed around Active Namenode & Standby Namenode for fail over process. At any point of time, you should not have two masters ( active Namenodes) at same time.

From Apache documentation link on HDFSHighAvailabilityWithQJM:

It is vital for the correct operation of an HA cluster that only one of the NameNodes be Active at a time. Otherwise, the namespace state would quickly diverge between the two, risking data loss or other incorrect results. In order to ensure this property and prevent the so-called “split-brain scenario,” the JournalNodes will only ever allow a single NameNode to be a writer at a time.

During a failover, the NameNode which is to become active will simply take over the role of writing to the JournalNodes, which will effectively prevent the other NameNode from continuing in the Active state, allowing the new Active to safely proceed with failover.

Zookeeper has been used to avoid Split - brain scenario. You can find role of Zookeeper in below question:

How does Hadoop Namenode failover process works?

Tobit