Which continuous integration server is able to queue jobs?

Question

Use case:

CI server polls some VSC repository and runs test suite for each revision. And if two or more revisions were commited, even in a relatively small time interval, I want the CI server to put each of them in queue, run tests for each, store the results, and never run tests again for those commits. And I don't want the CI server to launch jobs in parallel, to avoid performance issues and crashes in case of many simultaneous jobs.

Which CI server is able to handle this?

My additional, less important requirement is that I use Python and it is desirable to use software written in Python, so I looked at the Buildbot project, and I especially want to see reviews for this tool in the matter of is it usable in general and is it capable of replacing most popular solutions like Travis or Jenkins.

Kershaw

Answer 1

I have used jenkins to do this. (with subversion mainly, c/c++ build and also bash/python scripted jobs)

The easiest and default handling of VCS/SCM changes in jenkins is to poll for changes on a set time. A build is triggered if there is any change. More than one commit may be included in build (e.g. if 2 commits are done close together) when using this method. Jenkins shows links back to scm and scm update done as well as showing build logs and you can easily configure build outputs and test result presentation.

https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-Buildsbysourcechanges

What VCS/SCM are you using? Jenkins interfaces to a good few VCS/SCM: https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Sourcecodemanagement

This question answers how to make Jenkins build on every subversion commit: Jenkins CI: How to trigger builds on SVN commit

Watercraft

What do you mean by "More than one commit may be included in build"? – Kershaw

If two commits are close together they may be caught up together in one build. I mention the easiest method as it is the default operation, really very easy to make work. The answer on 'Jenkins CI: How to trigger builds on SVN commit' explains the easiest method and how to use subversion hook to get build triggered on every commit. – Watercraft

I consider this as a bad thing, because if tests failed you can't say which commit breaks them. – Kershaw

Two options are in answer 1. easiest but with this problem 2. use of svn hook described in other question and on jenkins subversion plugin docs – Watercraft

There is no option to push tasks in queue in Jenkins? – Kershaw

Jobs naturally have a queue in jenkins. Either with polling or hook solution jobs run one by one (they use the same workspace for checkout/build/test). – Watercraft

Answer 2

TeamCity is free (up to a number of builds and build agents) and feature-rich. It's very easy to install and configure, although it may take some time to find your way through the wealth of options. It is extremely well documented: http://www.jetbrains.com/teamcity/documentation/

It is written in Java but supports many tools natively and others through command-line execution, so you can build anything with it that you want. (I use it mostly for Ruby.) It understands the output of many testing tools; if you're not using one of them maybe yours can emulate their output. It's quite extensible; it has a REST API and a plugin API.

It can be configured to build on each commit, or to build all of the commits that arrived in a given time period, or to trigger in other ways. Docs here: http://confluence.jetbrains.com/display/TCD8/Configuring+VCS+Triggers

By default it starts a single build agent and runs one build at a time on that build agent. You can run more build agents for speed. If you don't want to run more than one build on a machine, only start one build agent on each machine.

Viscid

Answer 3

I dont want that CI server would launch jobs in parallel to avoid performance issues and crashes in cases of many simultanious jobs.

In buildbot you can limit the number of running jobs in a salve with max_build parameter or locks

Debutant

Related question: avoid that builders will run at the same time in buildbot – Debutant

Answer 4

As for Buildbot and Python, you may coordinate parallel builds by configuration, for example:

Modeling Parallel Processes: Steps

svn up
configure
make
make test
make dist

In addition, you can also try using a Triggerable scheduler for your builder which performs steps U,V,W.

From the docs:

The Triggerable scheduler waits to be triggered by a Trigger step (see Triggering Schedulers) in another build. That step can optionally wait for the scheduler's builds to complete. This provides two advantages over Dependent schedulers.

References:

• how to lock steps in buildbot
• Coordinating Parallel Builds with Buildbot

Preterite

As I understand, queuing is a default feature of Buildbot, as it stated on its webpage: "At its core, Buildbot is a job scheduling system: it queues jobs". Is it good tool at all, can you sahre more about your experience with it? I googled about Buildbot's integration with Pytest or PyUnit (unittest module in Python's standard library)s, but didnt found anything, how integration with test-frameworks is done in Buildbot? Can I store results for builds? – Kershaw

Answer 5

There is a Throttle Concurrent Builds Plugin for Jenkins and Hudson. It allows you to specify the number of concurrent builds per job. This is what it says on the plugin page:

It should be noted that Jenkins, by default, never executes the same Job in parallel, so you do not need to actually throttle anything if you go with the default. However, there is the option Execute concurrent builds if necessary, which allows for running the same Job multiple time in parallel, and of course if you use the categories below, you will also be able to restrict multiple Jobs.)

There is also Gitlab CI, a very nice modern Ruby project that uses runners to distribute builds so you could, I guess, limit the number of runners to 1 to get the effect you are after. It's tightly integrated with Gitlab so I don't know how hard it would be to use it as a standalone service.

www.gitlab.com

www.gitlab.com/gitlab-ci

To only run tests once for every revision you can do something like this:

1. build
2. post-build
   • check if the revision of the build is in /tmp/jenkins-test-run
   • if the revision is in the file skip tests
   • if the revision is NOT in the file run tests
   • if we ran the tests then write the ID in /tmp/jenkins-test-run

Encephalogram