Hazelcast: Questions regarding multi-node consistency

Question

(I could not find a good source explaining this, so if it is available elsewhere, you could just point me to it)

1. Hazelcast replicates data across all nodes in clusters. So, if data is changed in one of the nodes, does the node update its own copy and then propagate it to other nodes?

2. I read somewhere that each data is owned by a node, how does Hazelcast determine the owner? Is the owner determined per datastructure or per key in the datastructure?

3. Does Hazelcast follow "eventually consistent" principle? (When the data is being propagated across the nodes, there could be a small window during which the data might be inconsistent between the nodes)

4. How are conflicts handled? (Two nodes update the same key-value simultaneously)

Kierakieran

Answer 1

1. Hazelcast does not replicate (with exception of the ReplicatedMap, obviously ;-)) but partitions data. That means you have one node that owns a given key. All updates to that key will go to the owner and he notifies possible updates.

2. The owner is determined by consistent hashing using the following formula:

partitionId = hash(serialize(key)) % partitionCount

3. Since there is only one owner per key it is not eventually consistent but consistent whenever the mutating operations is returned. All following read operations will see the new value. Under normal operational circumstances. When any kind of failure happens (network, host, ...) we choose availability over consistency and it might happen that a not yet updated backup is reactivated (especially if you use async backups).

4. Conflicts can happen after split-brain when the split cluster re-merge. For this case you have to configure (or use the default one) MergePolicy to define the behavior on how conflicting elements are merged together or which one of both wins.

Eglanteen

Thank you for the response. I have a follow up question. Say, there is one node - node A. 1. Now another node B joins the cluster and inserts a map with some key-values. I assume B owns the map and the keys it has inserted. 2. Now B exits the cluster. Hazelcast will now replicate the keys in A, which is the only remaining node. 3. Now C joins the cluster and reads the keys. Hazelcast retrieves the key-values from A. Is this right? – Kierakieran

Since the partition table changes whenever a node joins / leaves the data will be re-migrated between the cluster nodes. So a node doesn't own his put keys but keys are always distributed. – Eglanteen