is it possible to lauch some oozie workflows with only one coordinator?

Question

I'm not sure to use the good tool for what I want.

I have a lot of workflows which can be dependent or not.

Exemple :

1. /workflow1
   • /workflow.xml
   • /job.properties
2. /workflow2
   • /workflow.xml
   • /job.properties
3. ....

I thought that we can have a corrdinator which can launch (with some data conditions) all the workflow. But I begin to think that is not the good practice.

Should we have one coordinator per workflow with all the conditions of executions + one bundle who launch all the coodinator ? like that :

1. /wf1
   • /workflow.xml
   • /job.properties
   • /coordinator.xml
2. /wf2
   • /workflow.xml
   • /job.properties
   • /coordinator.xml
3. /bundle.xml

OR one coordinator can launch all the workflow (they can be dependant or not) ?

1. /wf1
   • /workflow.xml
   • /job.properties
2. /wf2
   • /workflow.xml
   • /job.properties
3. /coordinator.xml

Rubino

Answer 1

It depends. If wf1 and wf2 are logically related, have the same frequency and have common dataset dependencies, you can put them by one coordinator (and run them simultaneously or one after another). But if they aren't than it's better to put them in separate coordinators.

You can launch several workflows from one using the sub-workflow feature:

<workflow-app name="root-workflow" xmlns="uri:oozie:workflow:0.4">
    <start to="run-wf1"/>
    <action name="run-wf1">
        <sub-workflow>
            <app-path>${appPath}/wf1.xml</app-path>
            <propagate-configuration/>
        </sub-workflow>
        <ok to="run-wf2"/>
        <error to="kill"/>
    </action>
    <action name="run-wf2">
        <sub-workflow>
            <app-path>${appPath}/wf2.xml</app-path>
            <propagate-configuration/>
        </sub-workflow>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

If you'd like to run them simultaneously than use forking.

Ellipsis

I have around 30 workflows, I dislike subworkflow because it's difficult to see quickly if all is ok in all the subworkflow. What's more, some of them depends on some same input but sometimes not or on other workflows... The scheduling will be difficult. It's the reason why I need to know what's the best way to do that. – Rubino

If you have 30 workflows you should definitely put them in different coordinators. I myself develop the system, which have around 50 coordinators, only 2 of them run more than one worfklow. – Ellipsis

OK, thanks. So do you use a bundle ? And how do you manage with the properties files ? Now I have one per workflow but when i try to use a bundle, the job.properties of the wf are not read. – Rubino

No, I don't use bundles. "the job.properties of the wf are not read" - you don't need them, you should define the workflow properties in a coordinator instead. – Ellipsis

Do you say that I can't have a property file per workflow/coordinator ? If it is the case, it will be complex for me to change all the properties name to put them in only one file :( – Rubino

I mean, if you have a coordinator wrapping a workflow, you don't need a property file for the workflow, cause you have one for the coordinator. – Ellipsis

I am ok with this too. But if I use a bundle which launch all the coordinators, i will not be able to keep one property file per coordinator, no ? – Rubino

I don't use bundles, I can't answer this question. – Ellipsis

Have I answered your original question? – Ellipsis