Things to consider when writing our own Continuous Integration server?

Question

I work on a large project in an organization that is (slowly) upgrading our development processes to be a bit more modern. We currently are considering moving to a Continuous Integration model; as part of this move, we are considering writing our own Continuous Integration server. We have a very mature (somewhat ossified) build process; we also have a large set of tests that we want to be run as build verification tests.

We have looked into several commercial CI servers, and it appears that the amount of work involved in customizing any of them to our individual needs is relatively high; so high that it may be worth our while to just custom write our own CI server. However, I feel that we may be missing some potential pitfalls of this process. The question of bugs in our implementation has been raised and considered; are there any other major considerations (other than the amount of effort involved in writing a CI system, of course) that we should keep in mind when evaluating our options? From anyone who has implemented a custom CI server, what were the particular troubles? And from anyone who has used a commercial CI system, were there any things that you wish you would have done yourself, or things which you were particularly happy you didn't need to do yourself?

Miles

Answer 1

I would strongly advise against this NIH thinking.

• Consider modifying an open source CI server like CruiseControl.NET
• Consider writing custom Nant tasks
• Consider modifying your project to be more amenable to existing options
• You don't want to be in the business of continuous integration, you want to just use it and see the benefits

Haha

Yes, thank you; can you give any specific reasons why rolling our own would be a bad idea so that I could pass those on to the team involved so we can make a rational decision? – Miles

* You don't want to be in the business of continuous integration, you want to just use it and see the benefits * You don't know how to do it * It's already been done well * Existing options have existing support communities * Existing options are being used in production on major projects and bugs have been discovered and fixed – Haha

@McWafflestix Time spent debugging your CI server takes away from the development you actually want to be doing. There are open source options that you can learn and may take some time, but you can have confidence that most of the bugs have been worked out. Once you have your CI server set up you don't want to have to fix things when you want to tweak it a little. – Disembody

You'll also get the community built around CruiseControl (or some other solution). That means guidance and sample scripts, etc. You can also recruit new devs that have knowledge of an existing CI system. I use CruiseControl.Net but it required a lot of trial and error to get running the first time. It's still a lot less time than building your own. – Termagant

Answer 2

Instead of a full-blown CI server, why not just build the customized pieces you need? Reuse as much as you can rather than have EVERYTHING custom, you can at least make some parts off-the shelf.

(By the way, I wouldn't characterize this as "NIH")

Note that even if you spend as much time configuring your builds to work with an existing CI server as you would building your own (probably being generous here) you then only need to maintain each custom bit - not the whole framework.

See what you can leverage, and work with it. Also see what tool seems to have promise for moving in the direction you are going. No CI server is going to work perfectly in your environment right out of the box. They all have to be tweaked and "met halfway".

Antecede

That's not a bad idea; it looks at it from a componentized perspective, which is certainly useful. – Miles

Answer 3

Doing this well would not be a small effort at all - the open source community is on at least the third generation of open source CI servers - with lots of lessons learned along the way.

I would certainly encourage you to look at one of the off-the-shelf CI servers that allows for easy extension, and build out those pieces that are missing. There is nothing in your question that isn't done (and done very well) in most of the existing CI servers.

I did implement a "poor man's" internal CI system prior to moving to an open source solution, and found that the open source systems were so much better, complete, and robust that my effort was pointless.

In particular, if you're looking for extensability, Hudson would be a great choice - it has a great plugin architecture, and the author is available for paid consulting work - probably a MUCH better use of your resources than rolling your own.

As an aside, prior to my involvement with the build process, a product build was a 14.5 hour affair, with no automated testing, packaging, or CM. Thanks to some effort, and following the best practices laid out by the existing CI community, the same project build is down to 7.5 minutes, with automated testing, packaging, and CM. In my experience, it's definitely worth some effort to work your build to comply with the existing best practices of the experienced CI community, rather than trying to make CI bend to your existing build.

Foremast

+1 for the ref to Hudson (and the wisdom that comes from trying to implement one in the past) – Palladio

Answer 4

Things I would consider:

• What kind of experience do we have on the team in this domain?
• Is the timeline such that the people on the team can put in the time to learn about the design of CI servers?
• Is there budget to hire people with this domain knowledge to supplement the team?
• Why do you think you can do this better than the existing solutions?

Walkin

Answer 5

I've been down this road as well. I've written an in-house automated build system 3 times now, for various companies that I have worked for. It's always been fun, and I'm the type of person who enjoys writing tools.

At the end of the day, though, it was never a professional system. It was always a system that was good enough, worked for us, and got us by. Unless of course, I went on vacation, or to a meeting in another city, etc. It got to the point that whenever I would go out of town the builds would fail, and fail, and fail. Not because of problems with our code, but because of little things that I would handle in the build system on a daily basis. And when I wasn't there to handle them, no one else knew what to do.

All of this took time away from our main focus - our own software development.

Finally, we have (at my urging) ditched the hand built perl script system that I wrote, and we've purchased a commercial system: Zed Builds and Bugs

Now, when something goes wrong it's a problem with our own code. The vendor's build system works. When we have questions, we ask them, or ask the community of users. When we need to understand how to do something, they have people to answer our questions.

And most importantly, I get to take vacations again :-)

I'm still the one handling most of the tasks with the build system, but it's not a question of writing it any more. It's a question of adapting our existing makefiles, Visual Studio projects, Ant build scripts, etc. to the new system.

Believe me, it's much easier to buy versus build in this case. Your core business isn't creating a build system (or you wouldn't have asked the question). Focus on your core business, and buy the tools you need for everything else.

Certifiable

Answer 6

I may try to give some hints from my point of view. At our company we were using the different configuration of the build environment and different target environment. By different configuration I mean different system architecture. And this is a point I would like to describe. (Although I do not know if this is your actual problem).

Build of the product was done on build env and to run tests and the product itself it has to be transfered to the target env. Thus it involved cross compiling. We didn't used a commercial or open source continuous integration solution, but we have used set of bash scripts (there was no time to create reasonable solution :( ) Running of the tests (doesn't matter if unit, intergration, sanity, components etc..) has to be done on target.

So the question you may ask are - what is development environment? Where would you like to run the tests? (We have been running test on target and small part of the test on the build env as well) Is target environemnt same as devel environment? Do you need to build for different configurations (platform, sw, etc..)?

You haven't stated what languages are you using. I know it may be irrelevant, but it is not :) (Even java behaves differently on different platforms (x86 vs x86-64)

Anything else like coverage of the code with test, some statical code analysis done during compilation (before..) running documentation task I consider as not very interesting as this tasks can be done as separate plugins.

Prosaic

I wasn't talking about management reasons.. just some technical. – Prosaic

Did you encounter any problems? Were there any particular successes? Was anything particularly difficult in the process? – Miles

Problems.. If you didn't considered this setup in the beginning it was really hard to instruct the build system & CI what to do and where to do it. Ad difficulties: Chain is like Source(Dev env) -> build(dev env) -> test (dev env) -> deploy (target env) -> test (target env) Problem is to instruct the CI you want to run test A on dev and on the target as well. The problem may be bigger if you need to execute the test differently on target platform. I repeat this was one particular problem we faced ( I consider it quite interesting). – Prosaic