Run tests affected by code changes first in TeamCity

Question

Using TeamCity, we want to run tests which are affected by code changes first.

NCrunch (http://ncrunch.net) and Mighty Moose (http://continuoustests.com/) provide solutions for that on a local development machine. Is there some solution for TeamCity?

Context: We are running many integration tests and programmers tests against our code base. It takes up to 4h to run all tests on the integration server. It runs on rather powerful hardware and there is not much room for improvement on that end. When a developer commits und pushes, it would be nice to get a quick feedback.

Hallux

Consider setting up TeamCity Agents on other servers. Another option is to set the integration tests to run nightly rather than on every check in. Unit tests would still run whenever relevant code is changed. You'll also want to set the build triggering to only fire when relevant code is changed. – Turkish

There are already 8 Agent servers... Also, only 5% of our tests are unit tests since this is a brown-field application that does not provide the needed level of separation and modularity for unit tests. – Impostume

@noah1989 - you might want to look at TypeMock or JustMock. – Turkish

@TrueWill: Yeah, I know, but unfortunately this doesn't help us much. We are stuck with old code (with lots of inline SQL) that no one dares to change. So we write tests around it and then refactor. We can't mock anything here, because we want to be sure the refactored (read: rewritten) code does the same thing as the old one did in SQL. – Impostume

@noah1989 - if you haven't read it, Working Effectively with Legacy Code targets your situation. – Turkish

Answer 1

To reduce overall tests duration, you probably can split your tests on two or even more parts and have separate build configuration in TeamCity for each part. Then you can create one more build configuration with snapshot dependencies on all these parts. Finally, you can add VCS trigger to this configuration with option to trigger on changes in dependencies. Build configurations actually running tests may not have triggers at all.

With this setup and enough agents you'll have several parts of your tests running in parallel. Note that snapshot dependencies results will be consistent, because when integration test build configurations are triggered, their revisions are fixed and will be the same for all triggered builds. The build configuration depending on test configurations may not have build steps at all, it's only task is to provide aggregated test results and triggering.

TeamCity also supports tests reordering for .NET and Java, you can read more about this feature in documentation: http://confluence.jetbrains.net/display/TCD65/Running+Risk+Group+Tests+First

Kvass

Thx for your answer. Most of the sugggestions we do already. (Splitting tests, different build configurations, triggering and pipelining builds.) – Hallux