Jenkins Partial Build / Modular Build on Commit Hook

Question

Tools:
Jenkins ver. 1.470
Maven 2
Subversion

Environment

Assume my build has a number of projects A-D. The dependency graph exists as shown. That is to say: B depends on classes in A, C depends on classes in B, D depends on classes in A. We create the jenkins builds such that they call the builds dependent upon them as a post-build action.

A
|--> B --> C
|--> D

Each night, we trigger a full build in Jenkins (A builds, triggers B (triggers C), triggers D). This is done easily enough by telling A to build nightly, and the rest cascades.

Problem

However, on a commit we want to build the projects that were committed to once.

• Situation 1: We poll the repository (or use commit hooks, it makes no difference) and find that there was a commit to B, then B will build and C will build. Success!

• Situation 2: We poll the repository and find that B and C were committed to in one commit, then Jenkins will try to build B (triggering a build of C), and build C (a second build). Failure. See what happens? C was built twice, taking up precious build time. Keep the build fast!

Does anyone know a way to only trigger the highest project in each committed build pipeline?

I suppose one solution would be a complex SVN hook that determines the highest project in each pipeline...

• Situation 3: Commit to B C and D in one commit. SVN hook finds C depends on B. The hook calls project-specific links to start builds for B and D.

Pitfalls: Very complex SVN commit hook. Have to upkeep the pipeline in the SVN hook.

I feel like this is a problem others have run into. Is there a Jenkins plugin that helps with this?

Pascale

In situation 2, the jenkins projects C & B are looking at the same svn project? – Fasano

Answer 1

It would be an idea to say jenkins to wait with the build until a build that c depends on is finished. The is an flag within the job configuration in order to do that. But you have to do this for each job. Btw ... there is also another flag that requires jenkins to wait with the build until a dependend job is finished.

Chuckchuckfull

Answer 2

I'm also looking for an efficient solution to this problem. I've seen several suggestions, but so far we've only managed to avoid one of the pitfalls by serializing the build using the Locks&Latches plugin. It doesn't prevent a project from building multiple times from a single checkin, but it will ensure that the project is rebuilt sequentially after the upstream project completes.

It's actually a complicated problem to solve in the general case, but I've been thinking of writing a plugin to deal with this. One simple solution is just to check if an upstream project is currently building, and remove yourself from the build queue if so. Since the upstream job will kick off your build when it's complete, this is one option.

A better option would be a plugin that automatically manages the build queue based on your dependency graph. This can be complicated, because you need to ensure that no build begins until all of its dependencies are completed. Essentially this means that every checkin will cause the plugin to automatically add all of the downstream builds to the queue so it can manage them. It's possible there's a clever, easier way to make this happen with the existing upstream/downstream triggers, but it's not clear to me just yet how to make it happen. There are build pipeline plugins already that claim to handle this situation, but they clearly do nothing to prevent the race condition where a downstream build can be triggered by the same checkin as an upstream build.

Czech