Error HBASE-ZOOKEEPER : Too many connections

Question

I am using Hbase-Hadoop combination for my application along with Data Nucleus as the ORM.

When I am trying to access Hbase via several threads at a single time, it throws exceptions as:

Exception in thread "Thread-26" javax.jdo.JDODataStoreException

org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

How can I handle this kind of situation? Is there any configuration I need to do to increase the connection pool?

Waynewayolle

Answer 1

Zookeeper servers have an active connections limit, which by default is 30. You need to increase this limit by setting maxClientCnxns property accordingly in your zookeeper config file, zoo.cfg.

For 100 connections:

maxClientCnxns=100

To tell zookeeper to impose no limit to the number of connections:

maxClientCnxns=0

Resume

Thanks for the response, I will get back to you once i try it out – Waynewayolle

where to find zoo.cfg? I cannot find it anywhere in the hbase directory – Calx

It's not inside hbase directory, it's inside zookeeper conf directory. If you are managing zookeeper using hbase, you can use the property hbase.zookeeper.property.maxClientCnxns in hbase-site.xml, but i don't recommend it for production environment, since it will not work in distributed mode. – Resume

Is it possible that I cannot find zookeeper directory? I use hbase so I have a hbase directory and subdirectories but zookeeper is nowhere. – Shalne

Yes, it's possible, you are likely using HBase to manage Zookeeper, so you have to use the property hbase.zookeeper.property.maxClientCnxns in hbase-site.xml, which is hbase conf file. It's fine for tests, but i don't recommend using it in a production environment. – Resume

Is there is any way to find out how many concurrent connections are active or open in zookeeper? – Broch

@JayaAnanthram Yes, you can use one of the "four-letter words" to inspect zookeeper connection info. zookeeper.apache.org/doc/trunk/… (If you need the number of connections, use "mntr" or "stat", else "cons" gives useful info about every open connection) – Rayshell

Answer 2

I don't suggest to blindly dump up the max connection, this config is not max connections to ZK, but max concurrent connections per client to ZK, according to the ZK document:

maxClientCnxns (No Java system property)

Limits the number of concurrent connections (at the socket level) that a single client, identified by IP address, may make to a single member of the ZooKeeper ensemble. This is used to prevent certain classes of DoS attacks, including file descriptor exhaustion. Setting this to 0 or omitting it entirely removes the limit on concurrent connections.

therefore, you must have more than 30 concurrent connections to ZK from the same IP address. in this case, use netstat to debug what that IP is, and inspect what could init those many connections at the same time.

Admiration

Answer 3

For the above issue you need to restart the master service of HBase. The command for the same is

sudo /etc/init.d/hadoop-hbase-master start

this needs to be done before entering to HBase shell.

Barbuto