HADOOP YARN - Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty

Question

I am evaluating YARN for a project. I am trying to get the simple distributed shell example to work. I have gotten the application to the SUBMITTED phase, but it never starts. This is the information reported from this line:

ApplicationReport report = yarnClient.getApplicationReport(appId);

Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = DEFAULT_PARTITION; AM Resource Request = memory:1024, vCores:1; Queue Resource Limit for AM = memory:0, vCores:0; User AM Resource Limit of the queue = memory:0, vCores:0; Queue AM Resource Usage = memory:128, vCores:1;

The solutions for other developers seems to have to increase yarn.scheduler.capacity.maximum-am-resource-percent in the yarn-site.xml file from its default value of .1. I have tried values of .2 and .5 but it does not seem to help.

Essa

how did you set your nano ~/hadoop/etc/hadoop/yarn-site.xml? – Finnougrian

did you also edit your ~/hadoop/etc/hadoop/mapred-site.xml accordingly? – Finnougrian

Hi @Essa can you please tell me how did you resolve this issue? – Olethea

Answer 1

Looks like you did not configure the RAM allocated to Yarn in a proper way. This can be a pin in the ..... if you try to infer/adapt from tutorials according to your own installation. I would strongly recommend that you use tools such as this one:

wget http://public-repo-1.hortonworks.com/HDP/tools/2.6.0.3/hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
tar zxvf hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
rm hdp_manual_install_rpm_helper_files-2.6.0.3.8.tar.gz
mv hdp_manual_install_rpm_helper_files-2.6.0.3.8/ hdp_conf_files
python hdp_conf_files/scripts/yarn-utils.py -c 4 -m 8 -d 1 false

• -c number of cores you have for each node
• -m amount of memory you have for each node (Giga)
• -d number of disk you have for each node
• -bool "True" if HBase is installed; "False" if not

This should give you something like:

Using cores=4 memory=8GB disks=1 hbase=True
Profile: cores=4 memory=5120MB reserved=3GB usableMem=5GB disks=1
Num Container=3
Container Ram=1536MB
Used Ram=4GB
Unused Ram=3GB
yarn.scheduler.minimum-allocation-mb=1536
yarn.scheduler.maximum-allocation-mb=4608
yarn.nodemanager.resource.memory-mb=4608
mapreduce.map.memory.mb=1536
mapreduce.map.java.opts=-Xmx1228m
mapreduce.reduce.memory.mb=3072
mapreduce.reduce.java.opts=-Xmx2457m
yarn.app.mapreduce.am.resource.mb=3072
yarn.app.mapreduce.am.command-opts=-Xmx2457m
mapreduce.task.io.sort.mb=614

Edit your yarn-site.xml and mapred-site.xml accordingly.

 nano ~/hadoop/etc/hadoop/yarn-site.xml
 nano ~/hadoop/etc/hadoop/mapred-site.xml

Moreover, you should have this in your yarn-site.xml

<property>
      <name>yarn.acl.enable</name>
      <value>0</value>
</property>

<property>
       <name>yarn.resourcemanager.hostname</name>
       <value>name_of_your_master_node</value>
</property>

<property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
</property>

and this in your mapred-site.xml:

<property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
</property>

Then, upload your conf files to each node using scp (If you uploaded you ssh keys to each one)

for node in node1 node2 node3; do scp ~/hadoop/etc/hadoop/* $node:/home/hadoop/hadoop/etc/hadoop/; done

And then, restart yarn

stop-yarn.sh
start-yarn.sh

and check that you can see your nodes:

hadoop@master-node:~$ yarn node -list
18/06/01 12:51:33 INFO client.RMProxy: Connecting to ResourceManager at master-node/192.168.0.37:8032
Total Nodes:3
     Node-Id         Node-State Node-Http-Address   Number-of-Running-Containers
 node3:34683            RUNNING        node3:8042                              0
 node2:36467            RUNNING        node2:8042                              0
 node1:38317            RUNNING        node1:8042                              0

This might fix the issue (good luck) (additional info)

Finnougrian

An useful setting is yarn.nodemanager.resource.detect-hardware-capabilities as true, see hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-common/… for more info (replace 3.2.0 in this URL with your Hadoop version). – Marplot

Answer 2

I got the same error and tried to solve it hard. I realized the resource manager had no resource to allocate the application master (AM) of the MapReduce application.
I navigated on browser http://localhost:8088/cluster/nodes/unhealthy and examined unhealthy nodes (in my case there was only one) -> health report. I saw the warning about that some log directories filled up. I cleaned those directories then my node became healthy and the application state switched to RUNNING from ACCEPTED. Actually, as a default, if the node disk fills up more than %90, YARN behaves like that. Someway you have to clean space and make available space lower than %90. My exact health report was:

1/1 local-dirs usable space is below configured utilization percentage/no more usable space [ /tmp/hadoop-train/nm-local-dir : used space above threshold of 90.0% ] ; 
1/1 log-dirs usable space is below configured utilization percentage/no more usable space [ /opt/manual/hadoop/logs/userlogs : used space above threshold of 90.0% ]

Cajole

Answer 3

Add below properties to yarn-site.xml and restart dfs and yarn

<property>
   <name>yarn.scheduler.capacity.root.support.user-limit-factor</name>  
   <value>2</value>
</property>
<property>
   <name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>
   <value>0.0</value>
</property>
<property>
   <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
   <value>100.0</value>
</property>

Thyroxine

Answer 4

Yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Yarn Node Manager Aux Service</description>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.scheduler.capacity.root.support.user-limit-factor</name>  
<value>2</value>
</property>
<property>
<name>yarn.nodemanager.disk-health-checker.min-healthy-disks</name>
<value>0.0</value>
 </property>
 <property>
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk- 
     percentage</name>
   <value>100.0</value>
    </property>
 </configuration>

mapred-site.xml

<configuration>
  <property>
     <name>mapreduce.framework.name</name>
     <value>local</value>
     <description>MapReduce framework name</description>
       </property>
</configuration>

This worked for me. This solution , bestowed upon me by my friend Yash

Retire

Thank you for contributing to the Stack Overflow community. This may be a correct answer, but it’d be really useful to provide additional explanation of your code so developers can understand your reasoning. This is especially useful for new developers who aren’t as familiar with the syntax or struggling to understand the concepts. Would you kindly edit your answer to include additional details for the benefit of the community? – Allonym