Oozie shell action memory limit

Question

We have an oozie workflow with a shell action that needs more memory than what a map task is given by Yarn by default.

How can we give it more memory?

We have tried adding the following configuration to the action:

<configuration>
  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>6144</value> <!-- for example -->
  </property>
</configuration>

We have both set this as an inline (in the workflow.xml) configuration and as a jobXml. Neither has had any effect.

Surmount

Answer 1

We found the answer:

A shell action is executed as an oozie "launcher" map task, and this task does not use the normal configuration properties.

Instead you have to prefix the properties with "oozie.launcher" to make them apply to the launcher task.

So in our case, if we use the following configuration for our shell action, it works.

    <configuration>
      <property>
        <name>oozie.launcher.mapreduce.map.memory.mb</name>
        <value>6144</value> <!-- for example -->
      </property>
    </configuration>

This is not obvious from the oozie documentation. We found this here: http://downright-amazed.blogspot.com/2012/02/configure-oozies-launcher-job.html

Surmount

Answer 2

thank your point, just edit workflow.xml file, add :

<workflow-app name="simple-ONE-wf" xmlns="uri:oozie:workflow:0.1">
    <start to='ONE'/>`enter code here
    <action name="ONE">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapreduce.map.memory.mb</name>
                    <value>4096</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-Xmx3200m</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                    <value>-XX:MaxPermSize=1g</value>
                </property>
                ...
            </configuration>
           ...
    </action>

    <kill name='kill'>
        <message>Something went wrong: ${wf:errorCode('firstdemo')}</message>
    </kill>
    <end name='end'/>
</workflow-app>

Evaporation

What's the value added by this answer compared to the (clearer, IMO) existing one posted 3 years ago? – Antrum

more details, it is very clearer, can just used, i meet perm OOM, only to add memory is no use, must to add opts and to set -XX:MaxPermSize=1g – Evaporation

Wow, if you're targeting Java 7 or earlier. Java 8 has been released even before this question has been asked. Please make it clear in your answer, explain how it's different and where/when it can be used. – Antrum

Thank you for this code snippet, which may provide some immediate help. A proper explanation would greatly improve its educational value by showing why this is a good solution to the problem, and would make it more useful to future readers with similar, but not identical, questions. Please edit your answer to add explanation, and give an indication of what limitations and assumptions apply. – Volumeter

the configuration needs to be at the action section. not at the workflow section. correct? – Hadst