Datanode not starts correctly

Question

I am trying to install Hadoop 2.2.0 in pseudo-distributed mode. While I am trying to start the datanode services it is showing the following error, can anyone please tell how to resolve this?

**2**014-03-11 08:48:15,916 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (storage id unknown) service to localhost/127.0.0.1:9000 starting to offer service
2014-03-11 08:48:15,922 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-03-11 08:48:15,922 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2014-03-11 08:48:16,406 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/in_use.lock acquired by nodename 3627@prassanna-Studio-1558
2014-03-11 08:48:16,426 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582) service to localhost/127.0.0.1:9000
java.io.IOException: Incompatible clusterIDs in /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode: namenode clusterID = CID-fb61aa70-4b15-470e-a1d0-12653e357a10; datanode clusterID = CID-8bf63244-0510-4db6-a949-8f74b50f2be9
    at**** org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:837)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:808)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:280)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:222)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
    at java.lang.Thread.run(Thread.java:662)
2014-03-11 08:48:16,427 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582) service to localhost/127.0.0.1:9000
2014-03-11 08:48:16,532 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-611836968-127.0.1.1-1394507838610 (storage id DS-1960076343-127.0.1.1-50010-1394127604582)
2014-03-11 08:48:18,532 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-03-11 08:48:18,534 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2014-03-11 08:48:18,536 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 

Doralynn

you might also benefit from one of my post... since I learned from my mistake other people could save time with the following link this should help to fix the correct VERSION file #35108945 – Faustinafaustine

Answer 1

You can do the following method,

copy to clipboard datanode clusterID for your example, CID-8bf63244-0510-4db6-a949-8f74b50f2be9

and run following command under HADOOP_HOME/bin directory

./hdfs namenode -format -clusterId CID-8bf63244-0510-4db6-a949-8f74b50f2be9

then this code formatted the namenode with datanode cluster ids.

Egalitarian

Great answer, this worked for me. I just wanted to note that you should run this command from the namenode (not from the problematic datanode). – Parable

Thanks for iceberg, but it seems everybody miss an important step to make the re-formation effect: restart the namenode. – Propertied

Thanks a lot for sharing this command, this helped for me to format Hortonworks hadoop cluster. But they have to provide a generic solution to format the namenode and spread the configs to datanode as Cloudera does. – Adjust

Another option in order to avoid formatting could also to change the VERSION file, and set clusterId to the correct value. My path to the file was /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current – Saiga

Appreciate the answer iceberg, the answer worked to help solve the problem in my case. – Bithynia

I am a little bit confused right now. Is it going to truncate all of my data on the namenode? format sounds scary. Could somebody please clarify it for me? – Dissimilitude

Anybody know why this is necessary? What's the reason for this random corruption ? – Undressed

If I format the HDFS I lose all the data, why would anybody do this? – Blizzard

I wish the Apache documentation weren't so obtuse with all this information! Thanks for your help!!! – Perse

Just wanted to mention that the clusterId from the datanode was used in hdfs namenode -format -clusterId CID-8bf63244-0510-4db6-a949-8f74b50f2be9 (executed on the namenode). – Proclaim

Answer 2

You must do as follow :

• bin/stop-all.sh
• rm -Rf /home/prassanna/usr/local/hadoop/yarn_data/hdfs/*
• bin/hadoop namenode -format

I had the same problem until I found an answer in this web site.

Pause

Hi Thanks for the timely help it worked correctly :-) – Doralynn

With pleasure :) , mark the answer as "accepted" so that other people can benefit from it ;) – Pause

I have no yarn_data folders on my pc – Grozny

@StepanYakovenko The folder is where the OP put his data node. You just need to figure out where you datanode's folder is. – Karalee

Answer 3

Whenever you are getting below error, trying to start a DN on a slave machine:

java.io.IOException: Incompatible clusterIDs in /home/hadoop/dfs/data: namenode clusterID= ****; datanode clusterID = ****

It is because after you set up your cluster, you, for whatever reason, decided to reformat your NN. Your DNs on slaves still bear reference to the old NN.

To resolve this simply delete and recreate data folder on that machine in local Linux FS, namely /home/hadoop/dfs/data.

Restarting that DN's daemon on that machine will recreate data/ folder's content and resolve the problem.

Tseng

Answer 4

Do following simple steps

• Clear the data directory of hadoop
• Format the namenode again
• start the cluster

After this your cluster will start normally if you are not having any other configuration issue

Farron

Thank you. This is what finally worked for me. – Sustain

Answer 5

DataNode dies because of incompatible Clusterids compared to the NameNode. To fix this problem you need to delete the directory /tmp/hadoop-[user]/hdfs/data and restart hadoop.

rm -r /tmp/hadoop-[user]/hdfs/data

Zoosporangium

Answer 6

I got similar issue in my pseudo distributed environment. I stopped cluster first, then I copied Cluster ID from NameNode's version file and put it in DataNode's version file, then after restarting cluster, its all fine.

my data path is here /usr/local/hadoop/hadoop_store/hdfs/datanode and /usr/local/hadoop/hadoop_store/hdfs/namenode.

FYI : version file is under /usr/local/hadoop/hadoop_store/hdfs/datanode/current/ ; likewise for NameNode.

Conchoidal

Pretty old post but still, thanks for saving my data. All other solutions require formatting the whole namenode which will evaporate all data. Your solution is much safer, thanks! – Dissimilitude

Answer 7

Here, the datanode gets stopped immediately because the clusterID of datanode and namenode are different. So you have to format the clusterID of namenode with clusterID of datanode

Copy the datanode clusterID for your example, CID-8bf63244-0510-4db6-a949-8f74b50f2be9 and run following command from your home directory. You can go to your home dir by just typing cd on your terminal.

From your home dir now type the command:

hdfs namenode -format -clusterId CID-8bf63244-0510-4db6-a949-8f74b50f2be9

Foretopmast

Answer 8

Delete the namenode and datanode directories as specified in the core-site.xml. After that create the new directories and restart the dfs and yarn.

Evvy

Answer 9

I also had the similar issue. I deleted namenode and datanode folders from all the nodes, and rerun:

$HADOOP_HOME/bin> hdfs namenode -format -force
$HADOOP_HOME/sbin> ./start-dfs.sh
$HADOOP_HOME/sbin> ./start-yarn.sh

To check the health report from command line (which I would recommend)

$HADOOP_HOME/bin> hdfs dfsadmin -report

and I got all the nodes working correctly.

Alfie

Answer 10

I had same issue for hadoop 2.7.7

I removed the namenode/current & datanode/current directory on namenode and all the datanodes

• Removed files at /tmp/hadoop-ubuntu/*
• then format namenode & datanode
• restart all the nodes.
• things work fine

steps: stop all nodes/managers then attempt below steps

1. rm -rf /tmp/hadoop-ubuntu/* (all nodes)
2. rm -r /usr/local/hadoop/data/hdfs/namenode/current (namenode: check hdfs-site.xml for path)
3. rm -r /usr/local/hadoop/data/hdfs/datanode/current (datanode:check hdfs-site.xml for path)
4. hdfs namenode -format (on namenode)
5. hdfs datanode -format (on namenode)
6. Reboot namenode & data nodes

Cooe

Answer 11

There's been different solutions to this problem, but I tested another easy solution and it worked like a charm :

So if someone get the same error, you just need to change the clusterID in the datanodes with clusterID of the namenode in the VERSION file.

With your case, here's were you can change it on datanode side :

namenode clusterID = CID-fb61aa70-4b15-470e-a1d0-12653e357a10; datanode clusterID = CID-8bf63244-0510-4db6-a949-8f74b50f2be9

• Backup the current VERSION : cp /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION.BK
• vim /home/prassanna/usr/local/hadoop/yarn_data/hdfs/datanode/current/VERSION and change

clusterID=CID-8bf63244-0510-4db6-a949-8f74b50f2be9

with

clusterID=CID-fb61aa70-4b15-470e-a1d0-12653e357a10

Restart the datanode and it should work.

Latini