Jenkins (Hudson) - Managing dependencies between parallel builds

Question

Using Jenkins or Hudson I would like to create a pipeline of builds with fork and join points, for example:

     job A
    /      \
 job B    job C
   |        |
 job D      |
    \      /
     job E

I would like to create arbitrary series-parallel graphs like this and leave Jenkins the scheduling freedom to execute B/D and C in parallel whenever a slave is available.

The Join Plugin immediately joins after B has executed. The Build Pipeline Plugin does not support fork/join points. Not sure if this is possible with the Throttle Concurrent Builds Plugin (or deprecated Locks & Latches Plugin); if so I could not figure out how. One solution could be to specify build dependencies with Apache Ivy and use the Ivy Plugin. However, my jobs are all Makefile C/C++/shell script jobs and I have no experience with Ivy to verify if this is possible.

What is the best way to specify parallel jobs and their dependencies in Jenkins?

Quartern

Answer 1

There is a Build Flow plugin that meets this very need. It defines a DSL for specifying parallel jobs. Your example might be written like this:

build("job A")
parallel (
    {
        build("job B")
        build("job D")
    },
    {
        build("job C")
    }
)
build("job E")

I just found it and it is exactly what I was looking for.

Pharmaceutical

Answer 2

There is one solution that might work for you. It requires that all builds start with a single Job and end with a definite series of jobs at the end of each chain; in your diagram, "job A" would be the starting job, jobs C and D would be the terminating jobs.

Have Job A create a fingerprinted file. Job A can then start multiple chains of builds, B/D and C in this example. Also on Job A, add a promotion via the Promotions Plugin, whose criteria is the successful completion of the successive jobs - in this case, C and D. As part of the promotion, include a trigger of the final job, in your case Job E. This can be done with the Parameterized Trigger Plugin. Then, make sure that each of the jobs you list in the promotion criteria also fingerprint the same file and get the same fingerprint; I use the Copy Artifact Plugin to ensure I get the exact same file each time.

Shepard

Thanks! I haven't tried the combination of the Promotions Plugin with the Parameterized Trigger. I'll give it a shot. Already using the Copy Artifact Plugin with fingerprinting. – Quartern

Works like a charm. The fingerprinting is required to mark job E as a downstream job of A. I did not realize the fingerprinting could be used for that. Highly appreciated. – Quartern