I am using ZooKeeper to store hierarchical data. In one test I stored more than 300K children for under one node. ZK client crashed due to a ConnectionLossException when I tried to retrieve ALL children.
Remodeling the data might solve the problem (e.g. by bucketing as it reduces the number of children per node).
However I am curious to know the answers of the following questions:
- ZK has a limitation of data that can stored on one node (no more than 1MB), is there a similar restriction on the number of children per node?
- Does increasing Java Heap size will allow ZK to scale out the number of children per node?
- Is there anyway to control the way these children are sent back to client? if the children are sent in one batch then a network "hiccup" could destroy the message and cause failure.
Thanks!