Hive/HBase Integration - Zookeeper Session Closes Immediately

Question

We have an 8 node cluster using CDH3u2 configured using Cloudera Manager. We have a dedicated master node running our only instance of zookeeper. When I configure hive to run local hadoop, executed from the master node, I have no problem retreiving the data from HBase. When I run distributed map/reduce via hive, I am getting the following error when the slave nodes connect to zookeeper.

HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default).

We have tried setting max connections higher (we even tried removing the limit). This is a development cluster that has very few users, I know that the problem is not that there are too many connections (I am able to connect to zookeeper from the slave nodes using ./zkCli).

Server side logs indicate that the session was terminated by the client.

Client side hadoop log says: 'Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

Any idea why I am unable to maintian a connection to zookeeper via Hive Map/Reduce?

Configs for hbase and zookeeper are:

# Autogenerated by Cloudera SCM on Wed Dec 28 08:42:23 CST 2011
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=1000
minSessionTimeout=4000
maxSessionTimeout=40000

HBase Site-XML is:

<property>
  <name>hbase.rootdir</name>
  <value>hdfs://alnnimb01:8020/hbase</value>
  <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
</property>

<property>
  <name>hbase.master.port</name>
  <value>60000</value>
  <description>The port master should bind to.</description>
</property>

<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
  <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>
</property>


<property>
  <name>hbase.master.info.port</name>
  <value>60010</value>
  <description>The port for the hbase master web UI Set to -1 if you do not want the info server to run.</description>
</property>
















<property>
  <name>zookeeper.znode.parent</name>
  <value>/hbase</value>
  <description>Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.</description>
</property>

<property>
  <name>zookeeper.znode.rootserver</name>
  <value>root-region-server</value>
  <description>Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.</description>
</property>


  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>The ZooKeeper client port to which HBase clients will     connect</description>
  </property>

  <property>
<name>hbase.zookeeper.quorum</name>
<value>alnnimb01.aln.experian.com</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".</description>

Gassy

Answer 1

Turns out that the Map/Reduce submitted by Hive is trying to connect to zookeeper at 'localhost', regardless of how the zookeeper.quorom is setup in the config file. I changed /etc/hosts to have to the alias 'localhost' point to the IP of my master node and the connection to zookeeper is maintained. Still looking for a better resolution, but this will work for now.

Gassy

Answer 2

I figured it out. It was a configuration issue (as I suspected all along). The solution was to:

-set ‘hbase.zookeeper.quorum’ within the ‘hive-site.xml’ and place it in the ‘hadoop-conf’ directory

What threw me off was that there is no 'hbase.zookeeper.quorum' in hive-default.xml. I had been playing with 'hive.zookeeper.quorum' which was not the correct configuration to change.

Gassy

Answer 3

I'm sorry for posting a new answer. I wanted to comment on the previous answer but the commenting UI seems to have disappeared >.< ...

Anyway, I wanted to say that I am experiencing the same problem, and it is solved by doing the /etc/hosts hack, but that seems like a very dirty solution...

Did anyone figure out a way of fixing this cleanly...??

Thanks :) !

Fireside

Answer 4

I meet exactly the same problem. What I did is to use the following conf to start hive cli and it works fine.

hive --hiveconf hbase.zookeeper.quorum={zk-host}

You should config HBase to use the external zookeeper and replace {zk-host} with the host of zookeeper. I'm still looking for how to resolve this when using jdbc to access hive.

Rhines