Can I make the SCM polling interval for Jenkins randomised

Question

In most Jenkins' examples the SCM poll value is designated */15 * * * *, ie. poll SCM every 15 minutes. That's fine when you have hundreds of jobs, but not if you have thousands of jobs as it results in thousands of request to the SCM (Subversion in this case) server at 15, 30, 45, and 0 minutes past the hour.

Is there some way to randomize the polling value in Jenkins so as to avoid the above scenario?

On the Jenkins master configuration screen there's a value Max # of concurrent polling. Should this be set (and to what value) to avoid the above scenario?

Calci

Late addition related to the issue, not the question: Jenkins allows you to limit the number of workers polling the SCM. This will resolve your issue without the need for randomization. See jenkins-ci.361315.n4.nabble.com/… – Redcap

Answer 1

Use H instead of * and Jenkins will randomly distribute the polling. Note that at present a good syntax has not been found for a frequency differing from once per hour/day/etc, so

H * * * *

will poll once per hour at a a pre-determined random minute.

H H * * *

will poll once per day at a pre-determined random hour and minute

H H H * *

will poll once per week

0 H * * *

will poll on the hour but once per day at a pre-determined random hour.

Keep in mind that you are allowed multiple cron lines and any will match, so until a good syntax for sub-hour frequency has been settled on, you can get closer (on average) with something like

H * * * *
H * * * *
H * * * *
H * * * *
H * * * *
H * * * *

will give you on average 15 minutes between polling (yes there will be 6 pollings per hour, but that is to give a good chance that at least one polling will fall in each quarter hour)

If you have good suggestions for a syntax distributing within the hour please respond to this thread:

https://groups.google.com/forum/?fromgroups#!jenkinsci-users/VghEjfygWuw/PuIG1o7u1GQJ%5B1-25%5D

Update (April 2013)

Jenkins 1.510 and newer includes a new syntax to allow specifying distributions within the hour

Kinematics

+1 for the 1.510 update. Too bad the latest LTS is 1.509.1. Almost enough of a reason to upgrade to a non-LTS release. – Toothy

Answer 2

If you set max# of concurrent polling to something between 1 and 3 then the polling requests will simply queue up and be processed serially (or at most 3 in parallel).

Given that all that is involved in the polling for SVN is effectively

svn info branch-url

They should all complete quickly, but at least you will have limited the parallel requests.

Calci

Answer 3

@Stephen Connolly's solution is likely a good solution for this.

If you really have thousands of jobs to worry about, consider setting the timing based on some part of the job name (Assuming your job names are reasonably distributed).

For example, if your job starts with 'B', set the timings to */2. If it starts with 'C', set it to */3 etc. ('A' would need something other than 1 though).

It's not a terrific solution, but if you're really looking at managing that many jobs it might be a work-around until you can find a better solution.

Chinfest

*/2 will run twice per hour, */3 will run three times per hour... I suspect you have the syntax the wrong way around, you're after – Kinematics

@StephenConnolly That's not correct; a */x indicates that there will be an interval of x units between jobs. The OP had it right: */15 * * * * is every 15 minutes, not 15 times per hour. – Soapberry

@edstaub the perils of SO on a phone ;-) – Kinematics

Answer 4

H H H * *

I think above will poll once per month not per week

Smash