Workflow error logs disabled in Oozie 4.2

Question

I am using Oozie 4.2 that comes bundled with HDP 2.3.

while working with a few example workflow's that comes with the oozie package, I noticed that the "job error log is disabled" and this makes debugging really difficult in the event of a failure. I tried running the below commands,

# oozie job -config /home/santhosh/examples/apps/hive/job.properties -run
job: 0000063-150904123805993-oozie-oozi-W

# oozie job -errorlog 0000063-150904123805993-oozie-oozi-W

Error Log is disabled!!

Can someone please tell me how to enable the workflow error log for oozie?

Silici

That -errorlog is new to V4.2, undocumented, and may have been de-activated in the HDP distro for many reasons. Including compatibility issues with other Hadoop components. So you probably would have to recompile the thing to activate it, at your own risk. By the way, what is it supposed to display that is not already in the -log output? – Mercurate

Thanks for the inputs ! The problem I am facing is when a workflow fails(I tried executing the example workflows that comes with the package) it is not giving me any information on what the error is, I couldn't find anything in the log to understand the root cause of the failure. Any leads on how this could be resolved will be of great help ! – Silici

Oozie logs show details of the coordination; the details of the actual execution are in YARN logs. Search the Oozie log for "External ID" and then find the matching entry in YARN web console. – Mercurate

Thanks Samson ! Will check that out and revert. – Silici

Hi Santosh, did you find a way to enable the error logs ? – Trilly

@Santosh did you get any way to check error log? – Lessielessing

Answer 1

In the Oozie UI, 'Job Error Log' is a tab which was introduced in HDP v2.3 on Oozie v4.2 .
This is the most simplest way of looking for error for the specified oozie job from the oozie log file.

To enable the oozie's Job Error Log, please make the following changes in the oozie log4j property file:

1. Add the below set of lines after log4j.appender.oozie and before log4j.appender.oozieops:

log4j.appender.oozieError=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.oozieError.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy
log4j.appender.oozieError.File=${oozie.log.dir}/oozie-error.log
log4j.appender.oozieError.Append=true
log4j.appender.oozieError.layout=org.apache.log4j.PatternLayout
log4j.appender.oozieError.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - SERVER[${oozie.instance.id}] %m%n
log4j.appender.oozieError.RollingPolicy.FileNamePattern=${log4j.appender.oozieError.File}-%d{yyyy-MM-dd-HH}
log4j.appender.oozieError.RollingPolicy.MaxHistory=720
log4j.appender.oozieError.filter.1 = org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.1.levelToMatch = WARN
log4j.appender.oozieError.filter.2 = org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.2.levelToMatch = ERROR
log4j.appender.oozieError.filter.3 =`enter code here` org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.3.levelToMatch = FATAL
log4j.appender.oozieError.filter.4 = org.apache.log4j.varia.DenyAllFilter 

2. Modify the following from log4j.logger.org.apache.oozie=WARN, oozie to log4j.logger.org.apache.oozie=ALL, oozie, oozieError

3. Restart the oozie service. This would help in getting the job error log for the new jobs launched after restart of oozie service.

Dalia

@Dalia i have used the code you have given but after adding these piece of code my Bundle job stuck with waiting status, am i doing something wrong – Lessielessing

i have added the code at the same place only in my oozie file log4j.logger.org.apache.oozie=WARN is as log4j.logger.org.apache.oozie=INFO, oozie, ETW, FullPIILogs was given and i have changed it to log4j.logger.org.apache.oozie=INFO, ALL, oozie, oozieError, ETW, FullPIILogs – Lessielessing

so please let me know if i am doing something wrong here – Lessielessing

Answer 2

As mentioned, the errorlog is new and may not be made available for good reasons. However it seems that you have the wrong expectation of the oozie error log.

The error log is meant to be a subset of the log file. Not an addition to it.

So yes, it could make things easier to debug, but if you checked the oozie log and did not find what you are looking for, the error log will not be the solution for you.

Probably you will want to look at the log of the underlying tasks, which can be found via the external ID.

Merovingian

my log just say Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [1] – Lessielessing

also i have check MapReduce job log and it is saying job has SUCCEEDED without any error, so can you help me what i am missing here. – Lessielessing

@Lessielessing If you need help and have looked at the available information on this site and in general, consider asking a new question. Comments are not the place for requests. -- At risk of stating the obvious: make sure to include all relevant information when you ask a question, otherwise it is very hard for people to help you. – Merovingian