Standard practices for logging in MapReduce jobs

Question

I'm trying to find the best approach for logging in MapReduce jobs. I'm using slf4j with log4j appender as in my other Java applications, but since MapReduce job runs in a distributed manner across the cluster I don't know where should I set the log file location, since it is a shared cluster with limited access privileges.

Is there any standard practices for logging in MapReduce jobs, so you can easily be able to look at the logs across the cluster after the job completes?

Moonfish

Answer 1

You could use log4j which is the default logging framework that hadoop uses. So, from your MapReduce application you could do something like this:

import org.apache.log4j.Logger;
// other imports omitted

public class SampleMapper extends Mapper<LongWritable, Text, Text, Text> {
    private Logger logger = Logger.getLogger(SampleMapper.class);

    @Override
    protected void setup(Context context) {
        logger.info("Initializing NoSQL Connection.")
        try {
            // logic for connecting to NoSQL - ommitted
        } catch (Exception ex) {
            logger.error(ex.getMessage());
        }
    }

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        // mapper code ommitted
    }
}        

This sample code will user log4j logger to log events to the inherited Mapper logger. All the log events will be logged to their respective task log's. You could visit the task logs from either JobTracker(MRv1)/ResourceManager(MRv2) webpage.

If you are using yarn you could access the application logs from command line using the following command:

yarn logs -applicationId <application_id>

While if you are using mapreduce v1, there is no single point of access from command line; hence you have to log into each TaskTracker and look in the configured path generally /var/log/hadoop/userlogs/attempt_<job_id>/syslog specified in ${hadoop.log.dir}/userlogs contains log4j output.

Corgi

thanks for the answer. I'm already using log4j, but I don't know which path should I specify for log output file (in log4j.properties file you need to specify log4j.appender.file.File property). I don't have access to any monitoring or admin web pages like ResourceManager page. Where would the log files be physically located in the cluster? Should I go to each Data Node to see the logs? – Moonfish

I have update the answer on how to access the logs from command line from both yarn and mapreduce v1. – Corgi

Answer 2

To add to @Ashrith 's answer: you can view the individual task tracker logs via the JobTracker GUI. The running task attempts are visible by the JT Gui and you can click on any of the following: stderr, stdout, and system logs. The system logs are where you find your log4j outputs.

Wildlife