HMaster process doesn't run on hadoop multi-node cluster after HBase installation

Question

I install a hadoop multi-node cluster on top of tow ubuntu virtual machines.

In next step I trying to installing HBase on this virtual cluster. But when I running HBase with start-hbase.sh, HMaster not running on master although HRegionServer has been run correctly on master and slave machines.

hbase-env.sh file contents:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC"
export HBASE_MANAGES_ZK=false

hbase-site.xml file contents:

<configuration>
    <property>
            <name>hbase.rootdir</name>
            <value>hdfs://machine-a:54310/hbase</value>
    </property>
    <property>
            <name>hbase.zookeeper.property.dataDir</name>
            <value>/usr/local/zookeeper</value>
    </property>
    <property>
            <name>hbase.cluster.distributed</name>
            <value>true</value>
    </property>
    <property>
            <name>hbase.zookeeper.property.clientPort</name>
            <value>2181</value>
    </property>
    <property>
            <name>hbase.zookeeper.quorum</name>
            <value>machine-a</value>
    </property>
</configuration>

zoo.cfg file contents:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zookeeper
clientPort=2181

Running zookeeper:

ZooKeeper JMX enabled by default 
Using config: /usr/local/zookeeper 3.4.13/bin/../conf/zoo.cfg
Starting zookeeper ... /usr/local/zookeeper 3.4.13/bin/zkServer.sh: line 140: ./zookeeper.out: Permission denied
STARTED

Running HBase:

/usr/local/hadoop-3.1.2/libexec/hadoop-functions.sh: line 2358: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution
/usr/local/hadoop-3.1.2/libexec/hadoop-functions.sh: line 2453: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.2/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
running master, logging to /usr/local/hbase-2.1.2/logs/hbase-saeed-master-machine-a.out
/usr/local/hadoop-3.1.2/libexec/hadoop-functions.sh: line 2358: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution
/usr/local/hadoop-3.1.2/libexec/hadoop-functions.sh: line 2453: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hbase-2.1.2/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
saeed@machine-b: running regionserver, logging to /usr/local/hbase-2.1.2/bin/../logs/hbase-saeed-regionserver-machine-b.out
saeed@machine-a: running regionserver, logging to /usr/local/hbase-2.1.2/bin/../logs/hbase-saeed-regionserver-machine-a.out

Yongyoni

Same problem here but not sure about the root cause. I rollback hbase version to 2.0.4. to make it work first. – Tigrinya

I think you need to delete the a redundant file in the CLASSPATH first. with sudo rm /usr/local/hbase-2.1.2/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar then see what happened. – Vc

Answer 1

I had a similar logs showing up when trying to start hbase. When I looked in the master logs could see the following:

2020-05-25 16:18:20,529 ERROR [master/VB-hbase:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master vb-hbase,16000,1590403688901: Unhandled exception. Starting shutdown. ***** java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.

I then followed the steps here

Apparently there might not be a direct link to the error we see for invalid variable logged during the start and master not coming up, as I could still see the error but all seems to work fine till now.

Suture

Answer 2

this maybe help(hbase 2.3.7)

add this line to /etc/profile( or ~/.bashrc) then source /etc/profile

export HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP=true

in $HBASE_HOME/bin/hbase, there are some code like this

#If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
# Allow this functionality to be disabled
if [ "$HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP" != "true" ] ; then
    HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH"  which hadoop 2>/dev/null)
fi

Hewlett