How can I reload oozie job configuration file without restart oozie job

Question

I'd like to know if there is a way to reload the configuration file of the oozie job without restart the oozie job ( coordinator ).

Because the coordinator actually runs many our tasks, maybe sometimes we only need change one line of the job configuration file, then make the update , without disturbing other tasks.

Thank you very much.

Allanite

Answer 1

The properties of oozie coordinator can be updated using below command once the coordinators start running. Update the property file in unix file system and then submit as below.

oozie job -oozie http://namenodeinfo/oozie -config job.properties -update coordinator_job_id

Note that all the created coordinator versions (including the ones in WAITING status) will still use old configuration. New configurations will be applied to new versions of coordinators when they materialize.

Danitadaniyal

Is there any way to make the WAITING workflows to use the new configuration? – Mercurialism

As of now there is no way to make WAITING coordinators to use new configuration. Once the coordinators in WAITING status complete their run, they can be rerun with updated configuration by using -refresh option as below. Rerunning a Coordinator Action or Multiple Actions Example: $oozie job -rerun <coord_Job_id> [-refresh] – Danitadaniyal

Answer 2

the latest oozie 4.1 allows to update coordinator definition. See https://oozie.apache.org/docs/4.1.0/DG_CommandLineTool.html#Updating_coordinator_definition_and_properties

Christoffer

Answer 3

Not really (well you could go into the database table and make the change but that might require a shutdown of OOZIE if your using an embedded Derby DB, and besides probably isn't advisable).

If you need to change the configuration often then consider pushing the value down into the launched workflow.xml file - you can change this file's contents between coordinator instantiations.

You could also (if this is a one time change) kill the running coordinator, make the change and start the coordinator up again amending the start time such that previous instances won't be scheduled to run again.

Carpal

Answer 4

Not really :-)

Here is what you can do.

1. Create another config file with properties that you want to be able to change in hdfs.
2. Read this file in the beginning of your workflow.

Example:

<action name="devices-location">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>

        <exec>hadoop</exec>
        <argument>fs</argument>
        <argument>-cat</argument>
        <argument>/path/to/config/file.properties</argument>

        <capture-output/>
    </shell>

    <ok to="report"/>
    <error to="kill"/>
</action>

<action name="report">
    <java>
        ...

        <main-class>com.twitter.scalding.Tool</main-class>

        <arg>--device-graph</arg>
        <arg>${wf:actionData('devices-location')['path']}</arg>

        <file>${scalding_jar}</file>
    </java>

    <ok to="end"/>
    <error to="kill"/>
</action>

Where the config file in hdfs at /path/to/config/file.properties looks like this:

path=/some/path/to/data

Christoffer

I found the answer very useful. But I have one query, in your approach , what if the property that is required to be mutable during runtime is jobTracker itself. would'nt shell action still point to old jobTracker and fail. – Luedtke