YARN Application exited with exitCode: -1000 Not able to initialize user directories

Question

I am getting:

Application application_1427711869990_0001 failed 2 times due to AM Container for appattempt_1427711869990_0001_000002 exited with exitCode: -1000 due to: Not able to initialize user directories in any of the configured local directories for user kailash
.Failing this attempt.. Failing the application.

I couldn`t find anything related to this exit code and the associated reason. I am using Hadoop 2.5.0 (Cloudera 5.3.2).

Unconventionality

I've also had this exit code -1000 error but no reason. Actualy reason: HDFS was corrupted and YARN container localization failed. Run hdfs fsck hdfs://xxx to verify this. – Inanna

Answer 1

Actually this is due to the permission issues on some of the yarn local directories. I started using LinuxContainerExecutor (in non secure mode with nonsecure-mode.local-user as kailash) and made corresponding changes. However due to some (unknown) reason NodeManager failed to clean local directories for users, and there still existed directories with previous user (in my case yarn).

So to solve this, I first had to find the value of the property yarn.nodemanager.local-dirs (with Cloudera use search option to find this property for YARN service, otherwise look into yarn-site.xml in hadoop conf directory), and then delate the files/directories under usercache for all the node manager nodes. In my case, I used:

rm -rf /yarn/nm/usercache/*

Unconventionality

Answer 2

Just in case if someone is lost with usercache location. If you don't have the yarn.nodemanager.local-dirs configured anywhere, look for it in the default location ${hadoop.tmp.dir}/nm-local-dir. Again, if the hadoop.tmp.dir is not configured in the core-site.xml, it will be under /tmp/hadoop-${user.name}. The user.name is the UNIX user that you are using to run the current Hadoop process. All the configuration file are under $HADOOP_INSTALL/etc/hadoop/ by default.

Licit

Answer 3

You need to apply this command

rm -rf /dn/yarn/nm/usercache/* { this is my configuration }

Please check you configuration inside YARN (MR2 Included) NodeManager Local Directories .

https://i.sstatic.net/I6KjG.jpg

You need to apply this to data nodes which reported error by the YARN

This is sample of my case

https://i.sstatic.net/kIuwA.jpg

ApplicationMaster reported that C90BFH04.localdomain:8042 , which is data node no 4 . So i applied only to the YARN directory in Node no 4

After that everything is OK !

Bernina

Answer 4

when i do the test of spark-submit-on-yarn and actor with cluster mode:

spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi /usr/local/install/spark-2.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.0.jar 100 As i am the same error:

Application application_1532249549503_0007 failed 2 times due to AM Container for appattempt_1532249549503_0007_000002 exited with exitCode: -1000 Failing this attempt.Diagnostics: java.io.IOException: Resource file:/usr/local/install/spark-2.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.0.jar changed on src filesystem (expected 1531576498000, was 1531576511000

fianlly,i fixed the error with set the property fs.defaultFS in the the $HADOOP_HOME/etc/hadoop/core-site.xml

Towe

Answer 5

corruptfileblocks files

hdfs fsck -list-corruptfileblocks

hdfs fsck -delete

Israel