Override hadoop's mapreduce.fileoutputcommitter.marksuccessfuljobs in oozie

Question

<property>
<name>mapreduce.fileoutputcommitter.marksuccessfuljobs</name>
<value>false</value>
</property>

I want to override the above property to true. The property needs to be false for the rest of the jobs on the cluster, but I need, in my oozie workflow, hadoop to create _SUCCESS file in the output directory after the completion of job. Its a hive action in the workflow which writes output. Please help.

Danaedanaher

Answer 1

Hive unfortunately overrides this capability by setting it's own NullOutputComitter:

conf.setOutputCommitter(NullOutputCommitter.class);

see

src/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
src/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java

Looks like you'll have to create the flag manually. We have filed HIVE-3700 for this.

Canicula

Is there a Hive Jira ticket for this? – Slither

I think issues.apache.org/jira/browse/HIVE-3700 is the ticket. I have added it to the answer. (There is no response on the ticket since 2012.) – Honestly

Answer 2

You can add 'dfs' command to your hive script, like

dfs -touchz '$table_base_path'/dt='${partition}'/_SUCCESS

https://archive.cloudera.com/cdh4/cdh/4/hive/language_manual/cli.html

Steiger