org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

Asked 11/6, 2013 at 7:46 Answered 13/8, 2022 at 10:31

I am trying to setup the multinode cluster of Hbase. When i do the jps on slave i get

5780 Jps
5558 HQuorumPeer
5684 HRegionServer
1963 DataNode
2093 TaskTracker

similarly on master i get

4254 SecondaryNameNode
15226 Jps
14982 HMaster
3907 NameNode
14921 HQuorumPeer
4340 JobTracker

EVerything is runnnig properly. But when i try to create table on hbase shell. It gives an error

ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

regionserver log of my slave(where region server is running):

2013-06-11 13:09:53,119 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$
2013-06-11 13:10:53,190 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:425)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1124)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)
2013-06-11 13:10:53,391 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at localhost,60000,137093$

FYI, i have also took care of /etc/hosts file on both master and slave.

127.0.0.1       localhost
127.0.0.1       naresh-PC

I again did changes in /etc/hosts file 127.0.1.1 to naresh-PC. But still getting this error

2013-06-11 14:51:17,781 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to Master server at naresh-pc,60000,137094$
2013-06-11 14:52:17,817 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to master. Retrying. Error was:
java.net.UnknownHostException: unknown host: naresh-pc
        at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:276)
        at org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:255)
        at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1111)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:138)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:2037)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2083)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:722)

Overmeasure answered 11/6, 2013 at 7:46 Comment(1)

Also, i am being able to do password-less ssh from master to slave and slave to master. – Overmeasure 11/6, 2013 at 9:28

Try clearing all the states in Zookeeper.

Stop Zookeeper
Wipe the Zookeeper data directory
Start Zookeeper

I was getting the same issue and followed this approach and it worked fine.

Bonheur answered 29/7, 2013 at 4:44 Comment(4)

Can you please explain how to perform these steps on Ubuntu. Can you list the commands that we need to follow. – Shelia 10/4, 2014 at 9:24

Ran into this today and wiping the zk data directory also worked for me. Nishu, your hbase-site.xml file should have a hbase.zookeeper.property.dataDir property that tells where this path is. Just make sure hbase is down before clearing it. – Lungan 28/2, 2015 at 21:39

It is enough just to delete /hbase znode in ZooKeeper via zkCli.sh with command rmr /hbase. No need to delete whole ZooKeeper directory. Be sure HBase Masters are not running during znode deletion. – Oneirocritic 17/5, 2019 at 15:31

On more recent versions of ZooKeeper, the rmr command has been removed, use deleteall instead. – Karyolysis 19/10, 2023 at 7:54

You need to change the configuration on the slave node to point at the master. It is currently pointing to localhost and not connecting to the actual master:

"org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: localhost/127.0.0.1:60000 at "

Munson answered 11/6, 2013 at 8:23 Comment(2)

well, i did give master's ip but still get the same problem. org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: master:60000 at – Overmeasure 19/6, 2013 at 8:50

After setting <property> <name>hbase.master</name> <value>ip-hmaster:60000</value> </property> in /usr/lib/hbase/conf/hbase-site.xml on all slave nodes the issue was solved. Thank you for the answer. – Scutellation 4/6, 2014 at 13:25

I'm hosting my own cluster inside Docker. Here's what worked in my case. I grepped the HBase log file for errors and found "Master passed us a different hostname to use"

`[root@docker-iop bin]# grep ERROR /var/log/hbase/hbase-hbase-regionserver-bi-mgmt01.local.log
2016-10-06 00:05:29,816 ERROR [regionserver/bi-mgmt01.local/111.11.2.3:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=my-host-name, but now=111.22.33.444'

I mapped my-host-name to 111.22.333.444 in my hosts file, restarted HBase and it worked.

Misplace answered 6/10, 2016 at 20:43 Comment(1)

I also had the same issue with a fully distributed hbase cluster with the configuration below.

Master Node (Node-A)
Backup Masters ($HBASE_HOME/conf/backup-masters) (Node-B & Node-C)
3 Replication servers (Node-A, Node-B & Node-C)

RCA: The backup-masters nodes were attempted to be started when the cluster started.

Solution I removed the backup masters by making $HBASE_HOME/conf/backup-masters empty in all hbase nodes.

So I had a cluster running without backup masters.

I wonder if the master node and master nodes must not also function as regionservers? The HBase documentation says otherwise though.

Soupy answered 13/11, 2016 at 14:47 Comment(0)

I came across the same issue and could not find anything, it turns out I was copy pasting from the Hbase documentation (https://hbase.apache.org/book.html#shell_exercises). I believe some character in there may be creating the error, so try to manually enter:

create 'test', 'cf'

Aeromechanics answered 26/10, 2021 at 9:22 Comment(0)

We resolved this issue. Solution is to

stop Hbase
log to zookeeper-client as root
execute command rmr /hbase-unsecure/meta-region-server
start Hbase

We stop/start Hbase through Ambari UI, delete /hbase... through server bash shell.

[root@s1 ~]# zookeeper-client
Connecting to localhost:2181
.......
[zk: localhost:2181(CONNECTED) 0] rmr /hbase-unsecure/meta-region-server

Tracitracie answered 21/1, 2022 at 3:43 Comment(0)

I use docker/docker-compose to set up my distributed hbase, after I make changes, I can not create table in hbase shell.

I docker rm all the related images, and rebuild them. It works. Also, simply rebuilding the images doesn't work...

Opheliaophelie answered 13/8, 2022 at 10:31 Comment(0)

Recommended topics

Hot tags