How to prevent certain Jenkins jobs from running simultaneously?

Question

I have a couple of jobs that use a shared resource (database), which sometimes can cause builds to fail in the (rare) event that the jobs happen to get triggered simultaneously.

Given jobs A through E, for example, is there any way to specify that A and C should never be run concurrently?

Other than the aforementioned resource, the builds are independent of each other (not e.g. in a upstream/downstream relation).

A "brute-force" way would be limiting number of executors to one, but that obviously is less than ideal if most jobs could well be executed concurrently and there's no lack of computing resources on the build server.

Frequency

Looks like this plugin is on the deprecated list :( – Botulin

Which plugin is on the deprecated list? – Acidulate

@marc.guenther: John's comment was related to Locks and Latches plugin mentioned in pwan's answer. It is proposed for deprecation here, but other than that I don't see any signs of it going away. So I guess you can achieve this using either 'Locks and Latches' or 'Throttle Concurrent Builds'. – Frequency

Avoid the Locks and Latches plugin. It is on the deprecation list for a reason. If you manually interrupt a job while it is waiting to acquire a lock, the job will not run anymore until the Jenkins master is restarted. The Throttle Concurrent Builds plugin is a much better option that does not compromise your Jenkins installation. – Suborn

Answer 1

There are currently 2 ways of doing this:

• Use the Throttle Concurrent Builds plugin.
• Set up those jobs to run on a slave having only 1 executor.

Nicotine

For some reason the simple approach of using only one executor didn't work for me. I configured it to 1, and 4 days later someone reported that there were 2 running,.. and the config was showing: max = 1. – Wilder

Does someone know if it is possible to use the Throttle Concurrent Builds to throttle two pipelines but allow at the same time that one of the pipelines executes its own jobs concurrently? For example, I have a pipeline A and a pipeline B, I do have A running and B won't be able because A has some job running. I want to be able to have N jobs of A running while none of B running if there is any of A doing it. – Fineness

Answer 2

The Locks and Latches plugin here should help.

This question is probably a dupe of How do I ensure that only one of a certain category of job runs at once in Hudson?

Glamour

Thanks, got it working! Quick reference: 1) define locks in Manage Jenkins -> Configure System -> Locks (in my case I used the name of the shared database); 2) set relevant jobs to use the locks: Configure -> Build Environment -> Locks – Frequency

My question indeed seems to boil down to the same thing as the older one. But I'd argue that this one is much clearer (no wall of basically irrelevant text in question body & no misleading tags)... – Frequency

Just a heads up - sometimes multiple jobs get the lock at the same time. You can avoid this by limiting the number of executors on that node. – Wojak

As @John T Dyer pointed out, Locks and Latches is on the proposed plugin deprecation list. Reading the comments on that page, it seems Throttle Concurrent Builds should be a viable replacement. – Frequency

For Jenkins: Throttle Concurrent Builds plugin – Bearable

Answer 3

That's an old question, but the topic can still be relevant, especially when running application tests on Jenkins.

The Lockable Resources Plugin allows you to define lockable resources that can be used by builds. If your build requires an resource, it takes the lock. If a second build requires the same resource (which then is already locked), it will be queued for the resource to be free.

Although the docs use computers or printers as examples for lockable resources, the database example from above should work as well.

In opposite to the Locks and Latches Plugin mentioned in answers from 2012, this package seems to be currently maintained (currently ~2016).

Implied

Answer 4

Have a look at the External Resource Dispatcher Jenkins plugin, which was first published in November 2012. This (relatively) new plugin seems to exactly cover this use case.

Chasechaser

It's full of bugs (I had to modify the source to get something even half-working) and no longer works with the latest Jenkins. Other than that it would have been exactly what I wanted :-) – Barrios

Answer 5

N.B. you don't need physical or virtual hardware for a slave/node, you can set up "slaves" that run on the master server.

Manage Jenkins > Manage Nodes > New node

and make a "dumb slaves" each with its own root directory.

Create a few slaves, execute them when the server boots, and then you have essentially created pools of executors.

You might have, say...

db - only one executor in your case. compile - limit according to hardware or # of CPUs. scripts - have many executors for all those little jobs that Jenkins is good at doing.

Excommunication

Answer 6

Old question, and whether this will work for your application I can't be sure as you didn't mention details of your application. However, I wanted to add the way that I handled this in our Rails application test suite.

Our application's database configuration (database.yml) isn't in the source repository. Instead, it lives in /var/lib/configs/uniquing_database.yml on the VM which runs our Jenkins instance.

One of the steps of our build process involves copying this config file to the project workspace:

cp /var/lib/jenkins/configs/myapp_unique_database.yml config/database.yml

and that config takes workspace and build number information exposed to the environment by Jenkins into account in order to create a uniquely named database for that job and it's specific execution:

test:
  adapter: postgresql
  encoding: unicode
  host: 127.0.0.1
  port: 5432
  database: myapp_test<%= ENV['JOB_NAME'].split('/').last %><%= ENV['BUILD_NUMBER'] %>

The rest of our build proceeds without any knowledge or care that it's running in a distinct database. Finally, at the end of our build, we make sure to drop that database so we don't have a bunch of test databases polluting the file system:

RAILS_ENV=test bundle exec rake db:drop

Leeward