ZooKeeper - adding peers dynamically?

Question

I'm new to ZooKeeper. This is what I need.

I've a network of peers.

1. At t=t_1 -> [peer-1 (Leader), peer-2] peer-1 is the master and all clients connect to this node.

2. At t=t_2 -> [peer-1 (Leader), peer-2, peer-3] At some later time peer-3 joins the group. Is it possible to add peer-3 to the list of zookeeper servers "dynamically" ( i.e., without restarting ZooKeeper on peer-1 ) ?

3. At t=t_3 -> [peer-3 (Leader), peer-4] After a while both peer-1 and peer-2 leave the group (e.g., die or are switched off.) Assuming that there is a way to dynamically add peer-3 and peer-4 to the group peer-3 becomes the leader and all client requests are send to peer-3.

Are there any other options that I can use apart from using ZooKeeper to do something like this.

thanks.

Unify

Answer 1

At the moment, you can't dynamically change the configuration of a zookeeper cluster without restarting. There is an open issue to fix this, ZOOKEEPER-107. The paper describing the cluster membership algorithm is quite interesting, and can be found here.

You can change the configuration of the cluster by restarting server nodes 1 at a time. For example, if you cluster has servers A,B,C, and you want to replace server C with D, then you can do something like,

• Bring down C
• Bring up D, it's peer list is A,B,D
• Take down B
• Change B's peer list to A,B,D
• Bring up B
• Take down A Change A's peer list to A,B,D
• Bring up A
• Change the client configuration of all clients to point to A,B,D

At t=t_1, you have a cluster with 2 zookeeper nodes. This is quite brittle, as if either node goes down, you will not be able to establish quorum (floor(N / 2) + 1), and the cluster will be unavailable. Generally zookeeper clusters are odd numbers.

I'm not sure what you are trying to do when you say,

peer-3 becomes the leader and all client requests are send to peer-3.

You can't specify which node in a zookeeper cluster is the leader, the nodes themselves will elect their leader, and leadership will change as nodes go up and down. As well, clients typically don't always connect to the leader, but clients are given list of machines in the cluster, and connect randomly to one, reconnecting if the server they are connected to goes down. You can set the leaderServes option to specify that the leader does NOT server client connections.

Lusatian

Looks like this is resolved in ZK alpha 3.5. – Comeback

zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html provides few details on how to dynamically re-configure the zookeeper. – Presidentelect

Answer 2

I would not suggest using the above for any production situation. The above solution only works if you are ok with losing ZK quorum for while until all changes are complete.

here's why: Bring down C Bring up D, it's peer list is A,B,D"

-> at this point A and B dont know about D -> D knows about A B

so at this point you have only A and B functioning in quorum next you take down B and you lose quorum.

you will lose access to zk data, until migration is complete and quorum is restored again. Most well designed apps using zk in this case failover to a readonly mode and will gracefully recover.

Until Zookeeper-107 is released under Zookeeper 3.5, you will need to choose you poison wisely.

Its better to :

1. just setup a new zk ensemble (zk cluster)
2. restore from snapshot
3. mirate apps from old zk ensemble to new zk ensemble
4. After migration is complete shutdown old zk ensemble

Meghanmeghann