buildbot vs hudson/jenkins for C++ continuous integration

Question

I'm currently using jenkins/hudson for continuous integration a large mostly C++ project. We have separate projects for trunk and every branch. Also, there are some related projects for the Java code, but the setup for those are fairly basic right now (we may do more later though). The C++ projects do the following:

• Builds everything with options for whether to reconfigure, do a clean build, or use a fresh checkout
• Optionally builds and runs all tests
• Optionally runs all tests using Valgrind's memcheck
• Runs cppcheck
• Generates doxygen documentation
• Publishes reports: unit tests, valgrind, cppcheck, compiler warnings, SLOC, open tasks, and code coverage (using gcov, gcovr, and the cobertura plugin)
• Deploys code nightly or on demand to a test environment and a package repository

Everything is configurable for automatic builds and optional for on demand builds. Underneath, there's a bash script that controls much of this, which farther depends on our build system, which uses automake and autoconf along with custom bash scripts.

We started using Hudson (at the time) because that's what the Java guys were using and we just wanted nightly builds. Since then, we've added a lot more and continue to add more. In some ways Hudson is great, but certainly isn't ideal.

I've looked at other solutions and the only one that looks like it could be a replacement is buildbot. Would buildbot be better for this situation? Is the investment worth it since we're already using Hudson? Why?

EDIT: Someone asked why I haven't found Hudson/Jenkins to be ideal. The short answer is that everything can be improved. I'm simply wondering if Jenkins is the best current solution for my use case or whether there is something better (buildbot?) that would be easier to maintain in the long run even as new requirements come up.

Pyrometallurgy

I haven't looked at Buildbot, but we do almost everything that you mention on multiple C++ projects on Hudson. What kind of non-ideal things do you see with Hudson/Jenkins? – Santiagosantillan

We've been very happy with Jenkins/Hudson so far. We've haven't really run into any cases where we felt it was inadequate or lacking. – Santiagosantillan

@SooWeiTan well for one, the UI.. – Brown

@Brown Having used Jenkins continuously since my last comment. I'm starting to agree. It gets even worse when you start installing plugins. We're pretty entrenched in the Jenkins ecosystem though it would be a big endeavour to switch systems. – Santiagosantillan

Voting to close as too broad. Even broader one: #26402 – Clitoris

@Brown Did any of you try buildbot and what did you think? – Helicon

Only a little discussion of Buildbot but good discussion on options for moving away from Jenkins news.ycombinator.com/item?id=19781646 – Helicon

Answer 1

Both are open source projects, but you do not need to change buildbot code to "extend" it, it is actually quite easy to import your own packages in its configuration in which you can sub-class most of the features with your own additions. Examples: your own compilation or test code, some parsing of outputs/errors to be given to the next steps, your own formating of alert emails etc. there are lots of possibilities.

Generally I would say that buildbot is the most "general purpose" automatic builds tools. Jenkins however might be the best related to running tests, especially for parsing and presenting results in nice ways (results, details, charts.. some clicks away), things that buildbot does not do "out-of-the-box". I'm actually thinking of using both to have sexier test result pages.. :-)

Also as a rule of thumb it should not be difficult to create a new tool's config: if the specification of what to do (configs, builds, tests) is too hard to switch from one tool to another, it is a (bad) sign that not enough configuration scripts are moved to the sources. Buildbot (or Jenkins) should only call simple commands. If it is simple to run tests, then developers will do it as well and this will improve the success rate, whereas if only the continuous integration system runs the tests, you will be running after it to fix the new code failures, and will loose its non-regression value, just my 0.02€ :-)

Hope it'll help.

Mannie

Thanks Christophe for the input and a good overview of the pros and cons of each. – Pyrometallurgy

"Buildbot (or Jenkins) should only call simple commands" - golden words – Osugi

Answer 2

The 'result integration' is also in jenkins/hudson, and you can relatively easily capture build products without having to 'copy them elsewhere'.

For our instance, the coverage reports and unit test metrics and javadoc for the java code is all integrated. For our C++ code, the plugins are a little lacking, but you can still get most of it.

we ran buildbot since pre 0.7, and are now running 0.8 and are only now seeing any real reason to switch, as buildbot 0.8 forgot about windows slaves for an extended period of time and the support was pretty poor.

Misplace

So do you recommend Jenkins or buildbot for a large C++ project? – Pyrometallurgy

Answer 3

There are many other solutions out there, besides Jenkins/Hudson/BuildBot:

• TeamCity by Jetbrains
• Bamboo by Atlassian
• Go by Thoughtworks
• Cruise Control
• OpenMake Meister

The specifics about what you are doing are not so important, in fact, as long as the agents (aka nodes) that you are doing them on support those tasks.

The beauty of a CI server is noticing when the build changes to trigger a new build (and test), publish the artifacts, and publish test results.

When you compare CI tools like those we mentioned, consider features like the usability of its interface, how easy is branching (and features it might offer like automatic merging), notifications (like XMPP/jabber), or an information-radiator (like hooking up a monitor to always show status). Product support is another thing to consider - Jenkins' support is only as good as who is responding to community questions at the time you have questions.

My personal favorite is Bamboo, but it comes with a license fee.

Bushnell

Thanks for the suggestions. In our case, we want to stick with FOSS solutions, which eliminates all of those options except Cruise Control. If you can offer a reason why I might want to switch to Cruise Control, that could be helpful. – Pyrometallurgy

Go for it: Jenkins is way better supported in the FOSS community than Cruise Control. Full steam ahead, man! – Bushnell

Go have actually become recently opensource too. – Quoth

Answer 4

I'm a long-time Jenkins user in the middle of evaluating Buildbot and would like to offer a few items for folks considering using Buildbot for multi-module solutions:

*) Buildbot doesn't have any out-of-the-box concept of file artifacts related to each build. It's not in the UI and it's not in any of the builtin "steps" modules as far as I can see:

http://docs.buildbot.net/current/manual/configuration/buildsteps.html

...and I see no third party plugin:

https://github.com/buildbot/buildbot/wiki/PluginList#steps

Buildbot does collect all the console output from a given build, but critically, you can't collect files related to it.

*) Given that artifacts are not supported, it's not easy to create "collector" projects that bring multiple modules into say, a single installer. Jenkins has a great feature that lets you parameterize a build with builds from other modules (the parameter type is a run).

*) Establishing dependencies between modules is trickier in Buildbot. Say you have a library that three binaries depend on, and you want those binaries to rebuild each time the library changes. Jenkins has triggers built into the UI. If you want to do triggers in Buildbot you have to script them using schedulers.Dependent, and it causes a lot of item congestion in the Schedulers UI.

*) When you're working in Buildbot, it seems that pretty much all of the configuration is done in master.cfg in code. This is awesome and frustrating.

*) Buildbot forces you to create a worker in addition to a master server. This is annoying for beginners and systems for which a single build server is sufficient.

My impression after two days of Buildbot evaluation is that we'll stick with Jenkins, primarily due to it having artifacts. Buildbot is a tool we'd only use if we had more extensive customization needs, and the time to do it.

Josephson

Answer 5

On the subject of buildbot and artifacts -- I don't have enough user score to make a comment -- you can get artifacts from buildbot 2.x series pretty easy with built-in file/directory upload actions. However you rarely want to just move files. Typically you make a triggered buildstep that does deployment directly off the worker for best results. eg push to cloud storage, containers, thirdparty (steam uploads), etc.

This way you can get metrics on the uploads and conditionally control them better (or even mix and match artifacts across worker machines).

Fibrilliform