Which version control programs can enforce running & passing of tests before integration of changes?

Question

At my work we currently use Aegis version control/SCM. The way we have it configured, we have a bunch of tests, and it forces the following things to be true before a change can be integrated:

• The full set of tests must have been run.
• All tests must have passed.

With test-driven development (TDD) these seem like sensible requirements. But I haven't heard of any way you can do this with any other version control systems. (We're not currently planning to switch, but I would like to know how to do it in the future without using Aegis.)

I would be interested in any VCS (distributed or not) that can do this, I'm also interested in any plugins/extensions to existing VCS that allow this. Preferably open source software.

ETA: OK, it seems like the usual thing to do is have VCS + continuous integration software, and running the tests is automated as part of the build, instead of as a separate step. If I understand correctly, that still lets you commit code that doesn't pass the tests, just you get notified about it -- is that right? Is there anything that would stop you from being able to integrate/commit it at all?

Weatherproof

Answer 1

IMO you're much better off using a continuous integration system like CruiseControl or Hudson if you want to enforce that your tests pass, and make the build rather than the check-in dependent on the tests results. The tools are straightforward to set up, and you get the advantages of built-in notification of the results (via email, RSS or browser plugins) and test results reporting via a Web page.

Regarding the update to the question, you're right - VCS + CI lets you commit code that doesn't pass the tests; with most CI setups, you just won't get a final build of your product unless all the tests pass. If you really want to stop anyone from even committing unless all the tests pass you will have to use hooks in the VCS as others have suggested. However, this looks to me to be hard to deal with - either developers would have to run all of the tests every time they made a checkin, including tests that aren't relevant to the checkin they are making, or you'd have to make some very granular VCS hooks that only run the tests that are relevant to a given checkin. In my experience, it's much more efficient to rely on developers to run the relevant tests locally, and have the CI system pick up on the occasional mistakes.

Cultivator

+1 Exactly, and what happens if your tests start taking a long time? People won't commit/checkin as often. – Quesenberry

I added to my question based on your suggestion. If you use continuous integration it still lets you integrate things that fail tests, you just find out about it quickly -- is that right? – Weatherproof

Answer 2

With subversion and git you can add pre-commit hooks to do this.

It sounds like you need to look at Continuous Intergration (or a variant of).

Think Git has a hook on apply patch too.

Cepeda

Just in theory, or can you point out any existing code/examples where people have done this? – Weatherproof

Answer 3

Subversion and git both support this via pre-commit hooks.

Visual Studio Team System supports this natively via checkin policies.

I believe that Rational ClearCase also supports it, though I've never seen that demonstrated so I can't say for certain.

Payola

Answer 4

We use git and buildbot to do something similar, though not quite the same. We give each developer their own Git repository, and have the buildbot set to build any time pushes to one of those repositories. Then there is someone who acts as the integrator, who can check the buildbot status, review changes, and merge their changes or tell them to fix something as appropriate.

There are plenty of variations of this workflow that you could do with Git. If you didn't want to have someone be the integrator manually, you could probably set the buildbot up to run a script on success, which would automatically merge that person's change into the master repository (though it would have to deal with cases in which automatic merge didn't work, and it would have to test the merge results as well since even code that merges cleanly can sometimes introduce other problems).

Spermary

Answer 5

I believe continuous integration software such as team city allow you to do pre-commit build and test. I don't know of any vcs that provides it directly...there may be some like the one you use but I'm not familiar with them.

Cracker

hm, cool. Team City calls this delayed commit: jetbrains.com/teamcity/delayed_commit.html But it's not open source, d'oh. – Weatherproof

Yeah, it's not open source but it is very nice to use. :-) it's definitely a product that is worth pitching to employers if they don't alreay have a cis. Sorry I didn't provide a link...was using an iPhone at the time...copy and paste won't be available until version 3. – Cracker

Answer 6

You can also use pre-commit hooks in Perforce. And, if you're a .NET shop, Visual Studio can be configured to require "gated" checkins.

Mcrae

Answer 7

VSTS with custom Work Items, right? I don't see anything wrong with using this. Built in reporting. The choice to automate. Why not?

Hedvah

Answer 8

What I do here is following a branch per task pattern which lets you test the code already submmitted to version control but still keeping the mainline pristine. More on this pattern here.

You can find more information about integration strategies here and also comments about Mark Shuttleworth on version control here.

Pasticcio

Answer 9

Most CI implementations have a mechanism to reject check-ins that don't meet all the criteria (most notably pass all the tests). They're called by different names. VCS should do what they do best.. version source code.

• TeamCity - Pre-tested commit
• TFS - Gated check-ins

Monopetalous

TeamCity's Pre-Tested Commit is not enforceable and its upto the developer to invoke (as of version 7) – Sprang