HBase standalone failed to connect (fail to create table)

Question

I am trying to deploy Hbase in standalone mode following this article: http://hbase.apache.org/book.html#quickstart. The version is 0.92.1-cdh4.1.2

But I am getting these errors when try to create a table:

Error message:

    13/04/01 14:07:10 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
13/04/01 14:07:10 WARN zookeeper.ZKUtil: hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)

Output log:

13/04/01 14:06:39 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
13/04/01 14:06:39 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 10231@localhost
13/04/01 14:06:39 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
13/04/01 14:06:39 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/04/01 14:06:39 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

My configurations:

1. Added JAVA_HOME in hbase-env.sh

2. hbase_site.xml

   <name>hbase.rootdir</name>
   
   <value>file:///home/hadoop/data</value>
   

   <name>hbase.zookeeper.property.dataDir</name>
   
   <value>file:///home/hadoop/zookeeper</value>
   

I have tried to modify /etc/hosts, which looks likes this (oracle is the hostname):

127.0.0.1   localhost oracle

or

[server ip]   localhost oracle

But these do not work for me.

My exact codes are:

[usr@oracle bin]$ ./start-hbase.sh 
starting master, logging to /home/hadoop/hbase-0.94.6/bin/../logs/hbase-usr-master-oracle.out
[usr@oracle bin]$ hbase shell
13/04/01 14:57:55 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.92.1-cdh4.1.2, rUnknown, Thu Nov  1 18:02:07 PDT 2012

hbase(main):001:0> create 'test','cf'

Thanks in advance!

Yahoo

try to open hbase web ui in any browser: localhost:60010 Does it work? – Selma

Also, is this directory structure exist? – Selma

@Sourav yes they all work, still could not get it to work. I have switched to use the cloudera bundle and it is much easier. – Yahoo

I have this error but can't use the Cloudera bundle and so would still love some help. I have a non-existent ZooKeeper dataDir as the hbase quickstart advises (hbase will create one) and have removed all ipv6 references in my /etc/hosts file. – Cacomistle

@Hypnos: Are you able to find a solution to this problem. I am also stuck here. – Xhosa

@Inquisitive no I never solved it, I just switched to the cloudera bundle and never looked back – Yahoo

Answer 1

Looks like you are missing hbase.zookeeper.quorum in your configuration. Please check this HBase guide chapter, it may help: http://hbase.apache.org/book/zookeeper.html

Also please check zookeeper is started on right port and (what happens more often) correct IP interfaces.

Elvinelvina

Using the standalone hbase I get the same error so zookeeper is launched by hbase as needed. When the table is created hbase can't launch zookeeper because it says "ZooKeeper exists failed after 3 retries". Which seems to say that hbase thinks zookeeper exists but the connection fails. I look at jps -l and see no zookeeper so why does it think it exists? – Cacomistle

Answer 2

I'm not sure if you're on Mac, but I had this problem as well. It turns out that it was because a .DS_STORE file was in a folder of -ROOT- region server and causing a connect exception. Very strange!

Pearlypearman

can you explain what "a folder of -ROOT- region server" means. The default standalone config uses a version of ZooKeeper launched by Hbase itself. I get this error on the mac with the standalone Hbase. – Cacomistle

Answer 3

In HBase 1.2.3 I've got almost the same error "ZooKeeper exists failed after 4 retries" in the standalone mode. It was caused by running ./start-hbase.sh without having permissions to connect to the port 2181. The solution turned out to be really simple:

sudo ./start-hbase.sh

Just in case, the configuration of hbase-site.xml is:

<configuration>
   <property>
      <name>hbase.rootdir</name>
      <value>file:///home/hadoop/HBase/HFiles</value>
   </property>
   <property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/home/hadoop/zookeeper</value>
   </property>
</configuration>

Drusie

this actually solved it for me somehow. It was driving me nuts, very strange that this is not in the instructions as it does seem necessary in macOS Big Sur. – Triptych

Answer 4

Something might be related to your configuration. At first, I download HBase tar(http://hbase.apache.org/book/quickstart.html) to run HBase. I didn't follow the link to do any configuration, just download, start HBase, run "HBase shell". But when creating a table, it gave me error like you mentioned. And then, I search HBase installation on Mac, since I use Mac. I adopt to use "brew install hbase" to install HBase(http://chase-seibert.github.io/blog/2013/02/01/getting-starting-with-hbase-and-pig.html). After finished installation, I re-create the table. Success.

So re-consider your error, it must be linked with your JAVA_HOME or other configurations on HBase. Finding the answer from this way might help you.

Paradis

Answer 5

The error "ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries" most probably indicates you don't have zookeeper running - Before staring the HBase shell you can verify if the Zookeeper quorum is up, using:

$ jps

the command will list all the java processes on the machine i.e. the probable output has to be - for the standalone HBase setup you're trying to setup (ignore the numbers in the left column i.e. pid):

62019 Jps
61098 HMaster        
61233 HRegionServer     
61003 HQuorumPeer

If your output doesn't have the HQuorumPeer - indicates that the zookeeper isn't running.

Zookeeper is required for HBase cluster - as it manages it.

---

Solution:

In your HBase directory, first stop the HBase:

$ ./bin/stop-hbase.sh

If you're trying to work out the 'standalone HBase' example - stick to the minimum conf provided in the example:

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///home/adio/workspace/hadoop/hbase/directories/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/adio/workspace/hadoop/hbase/directories/zookeeper</value>
  </property>
</configuration>

i.e. your conf/hbase-site.xml should have the above content.

Once set, now start the HBase again:

$ ./bin/start-hbase.sh

P.S. anyone if going through these steps still have the issue not resolved - leave your issue in the comment section.

---

Few relevant answers: 1 2 3

Loop

Answer 6

stop the hbase by stop-hbase.sh locate to your zookeeper folder and empty the contents of the folder. then restart the hbase and hbase shell. which solved the issue for me

Latchstring

The first time launch of a standalone hbase will create the folder according to the hbase quickstart. I've tried with a clean dataDir and with a non-existent one, same error. – Cacomistle

When I do this, it fixes the ZK error, but I also lose my local HBase tables. – Fulmar

Answer 7

I also ran into this issue while setting up hbase. A couple minutes later I figured that I had forgotten to start hbase by running the following command :

hbase-1.1.2/bin/start-hbase.sh

After running this command, hbase shell worked just fine.

Shebeen

Answer 8

It seems there is some error in hbase-site.xml conf. According to the quickstart document, standalone HBase hbase-site.xml should like this:

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///home/testuser/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/home/testuser/zookeeper</value>
  </property>
</configuration>

And there is no file:// in Zookeeper configuration. And take care when you have failed to start HBase, you have to delete files you store hbase and zookeeper. According to the above configuration, you have to delete all files under /home/testuser/hbase and /home/testuser/zookeeper. And try to check the log error info. I have fixed this error when I have the right conf and add JAVA_HOME variable in hbase-env.sh.

Zipah

Answer 9

Edit /etc/hosts file ... in second line change 127.0.1.1 to 127.0.0.1 then start Hbase... it should work fine. If not working... try steps on this link http://lets-do-something-big.blogspot.in/2014/01/hbase-installation-on-ubuntu-lubuntu.html

Percussive

No luck with that myself. Also took out all ipv6 references to localhost in /etc/hosts – Cacomistle

Answer 10

I too had the same issue.set the owner and group properly in both folders hbase.rootdir and hbase.zookeeper.property.dataDir.Give 755 permission to both folder.

sudo chown -R hadoop_user:hadoop_user_group home/hadoop/data
sudo chmod -R 755 home/hadoop/data

Vosges

Answer 11

The problem lies in 'hbase-site.xml' under the 'hbase.zookeeper.property.dataDir' property

  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/usr/lib/hbase/zookeeperdata</value>
  </property>

Note: Under this property set path in 'LOCAL' file system.

We only need to specify the directory on the local filesystem where HBase and ZooKeeper write data. For example in this case after you perform 1) start-hbase.sh 2) hbase shell command

Navigate to the path(In my case /usr/lib/hbase/zookeeperdata), where you will see a file named myid.

Summarizing the validation points

1) On performing jps following proccesses should run HQuorumPeer,ResourceManager,HMaster,NameNode,Main,HRegionServer,SecondaryNameNode,DataNode,Jps,NodeManager

2) Under hbase-site.xml, for the property 'hbase.zookeeper.property.dataDir', the path should be set to local path i.e. the folder should be locally present.

3) After performing start-hbase.sh and hbase shell command--> Navigating to the path specified in hbase.zookeeper.property.dataDir(IN my example the path is /usr/lib/hbase/zookeeperdata) , a file named myid should be present.

Mead

Answer 12

(1) Just run ./$HBASE_HOME/bin/start-habse.sh, below exception will disappeared.

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:419)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:905)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:648)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

 - List item

Cerebrate

Please, be more careful when formatting your answer. – Gonzalez