Applications not shown in yarn UI when running mapreduce hadoop job?

Question

I am using Hadoop2.2. I see that my jobs are completed with success. I can browse the filesystem to find the output. However, when I browse http://NNode:8088/cluster/apps, I am unable to see any applications that have been completed so far ( I ran 3 wordcount jobs, but none of it is seen here).

Are there any configurations that need to be taken into account?

Here is the yarn-site.xml

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>NNode</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
<!--
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
-->

Here is mapred-site.xml:

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

I have job history server running too:

jps
4422 NameNode
5452 Jps
4695 SecondaryNameNode
4924 ResourceManager
72802 Jps
5369 JobHistoryServer

Aegean

Try to check hadoop logs files. you might find warnings or errors there. But it seems that your datanode is down – Roose

@eliasah: The datanode is running fine. I did not show jps on datanode. what I showed above is namenode(master). The datanode runs fine and even my jobs are finished. – Aegean

@brainstorm Can you post the entire output of a wordcount job? All of it. – Psalmbook

And by output, I mean what you get in the terminal – Psalmbook

@climbage: Thanks for helping out. Two things I observed, one is I need to start yarn in my namenode. Then it seems to work. If I do start-yarn.sh in my client, it does not start resourcemanager. I dont know why. Secondly, Although I see my job being listed in application window, I cannot find information about number of mappers and reducers etc. In hadoop1, it was easy to find them in 50030 port I guess. Do you know how to get that info, in hadoop2 on the UI – Aegean

Answer 1

After applications are completed, their responsibility might be moved to Job History Server. So check Job History Server URL. It normally listen on port 19888. E.g.

http://<job_history_server_address>:19888/jobhistory

Log directories and log retain durations are configurable in yarn-site.xml. With YARN, even one can aggregate logs to a single (configurable) location.

Sometimes, even though application is listed, logs are not available (I am not sure if its due to some bug in YARN). However, almost each time I was able to get the logs using command line:

yarn logs -applicationId the_application_id

Athough there are multiple options. Use help for details:

yarn logs --help

Suborbital

Answer 2

you can refer Hadoop is not showing my job in the job tracker even though it is running

                    conf.set("fs.defaultFS", "hdfs://master:9000");
                    conf.set("mapreduce.jobtracker.address", "master:54311");
                    conf.set("mapreduce.framework.name", "yarn");
                    conf.set("yarn.resourcemanager.address", "master:8032");

I tested in my cluster. It works!

Talkingto