How do I tell MSTEST to run all test projects in a Solution?

Question

I need to know how to tell MSTEST to run all test projects in a solution file. This needs to be done from the command line. Right now I have to pass it a specific project file, I'm trying to get it to run from a SOLUTION file.

I'm hoping this is possible, because in Visual Studio, hitting Ctrl+R, A, runs ALL tests in the currently opened solution.

The way I've interpretted the help files, you have to pass in each DLL specifically.

I want to run this from the command line from my CruiseControl.NET server, so I can write other utilities to make this happen. If there is a wierd way of getting this to happen through some OTHER method, let me know.

How do I tell MSTEST to run all test projects for a solution?

<exec>
    <!--MSTEST seems to want me to specify the projects to test -->
    <!--I should be able to tell it a SOLUTION to test!-->
    <executable>mstest.exe</executable>
    <baseDirectory>C:\projects\mysolution\</baseDirectory>
    <buildArgs>/testcontainer:testproject1\bin\release\TestProject1.dll 
    /runconfig:localtestrun.Testrunconfig 
    /resultsfile:C:\Results\testproject1.results.trx</buildArgs>
    <buildTimeoutSeconds>600</buildTimeoutSeconds>
</exec>

Hanafee

Did you ever get this resolved? I cant figure out how to make the "CreateItem" stuff in CC.NET? – Inhalant

Answer 1

To elaborate on VladV's answer and make things a bit more concrete, following the suggested naming convention running your tests can be easily be automated with MSBuild. The following snippet from the msbuild file of my current project does exactly what you asked.

<Target Name="GetTestAssemblies">
    <CreateItem
        Include="$(WorkingDir)\unittest\**\bin\$(Configuration)\**\*Test*.dll"
        AdditionalMetadata="TestContainerPrefix=/testcontainer:">
       <Output
           TaskParameter="Include"
           ItemName="TestAssemblies"/>
    </CreateItem>
</Target>
<!-- Unit Test -->
<Target Name="Test" DependsOnTargets="GetTestAssemblies">
    <Message Text="Normal Test"/>
<Exec 
    WorkingDirectory="$(WorkingDir)\unittest"
    Command="MsTest.exe @(TestAssemblies->'%(TestContainerPrefix)%(FullPath)',' ') /noisolation /resultsfile:$(MSTestResultsFile)"/>
    <Message Text="Normal Test Done"/>
</Target>

Furthermore integrating MsBuild with CruiseControl is a piece of cake.

Edit
Here's how you can 'call' msbuild from your ccnet.config.

First if you do not already use MSBuild for your build automation add the following xml around the snippet presented earlier:

<Project DefaultTargets="Build"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    ..... <insert snippet here> .....
</Project>

Save this in e.g. RunTests.proj next to your solution in your source tree. Now you can modify the bit of ccnet.config above to the following:

<msbuild>
  <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
  <workingDirectory>C:\projects\mysolution\</workingDirectory>
  <baseDirectory>C:\projects\mysolution\</baseDirectory>  
  <projectFile>RunTests.proj</projectFile>
  <targets>Test</targets>
  <timeout>600</timeout>
  <logger>C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
</msbuild>

Averment

How and where do you incorporate this into CruiseControl config? Does this code you provided need to reside in an NANT task .build script? Or does this XML go directly into CC.NET config? – Inhalant

I wish it was a "piece of cake". I think its a P.O.S. personally! How do you do this in CruiseControl.NET? The original question didnt ask how to do this in MSBuild! – Inhalant

@Devtron expanded the answer, does this help? – Averment

Cheers! I really appreciate this help! I had a feeling I would need to use a Project file to do this...Thank you. – Inhalant

In the MSBuild script, does the Target need to specify "GetTestAssemblies"? How does MSTest know to use "GetTestAssemblies"? – Inhalant

The name of the target is irrellevant. It is used as a precondition for the execution of mstest.exe by the Test target, GetTestAssemblies is listed in the DependsOnTargets attribute. – Averment

Cool. It seems to be trying to work. Its now stuck on the EXEC command for MSTest.exe path. It cant find it, even if I give it an explicit path. Why is there no path for: <EXEC Command="MSTest.exe.."/>? I can see in my build log that all the correct DLLs are being supplied. It just cant find MSTest.exe, even with the full path... – Inhalant

Answer 2

This is an old thread, but I have been struggling with the same issue and I realized that you can really just run MSTest on every dll in the whole solution and it doesn't really cause any problems. MSTest is looking for methods in the assemblies marked with the [TestMethod] attribute, and assemblies that aren't "test" assemblies just won't have any methods decorated with that attribute. So you get a "No tests to execute." message back and no harm done.

So for example in NAnt you can do this:

<target name="default">
    <foreach item="File" property="filename">
        <in>
            <items>
                <include name="**\bin\Release\*.dll" />
            </items>
        </in>
        <do>
            <echo message="${filename}" />
            <exec program="C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe">
                <arg value="/testcontainer: ${filename}" />
                <arg value="/nologo" />
            </exec>
        </do>
    </foreach>
</target>

and it will run all the test methods in every dll in every bin\Release folder in the solution. Those which are not test dlls will return a "No tests to execute." and those that have tests will have the tests run. The only part I haven't figured out yet is that (in NAnt) execution stops the first time a command returns a non-zero value. So if any unit tests fail it doesn't keep going to execute any tests in subsequent assemblies. That is not great, but if all the tests pass, then they will all run.

Hindemith

btw - the way to get NAnt to keep running even if some unit tests fail is to put the failonerror="false" attribute on the exec task. That way all your tests will run even if some in one project fail. You can store the return values of each test run in a property and then after all the tests have, you can check that property and if it is greater than 0 you can fail it manually like this: <fail if="${int::parse(result) > 0}" message="Some tests failed." /> – Hindemith

Answer 3

I just resolve this problem recently. Here is my proposal: Use testmetadata + testlist option of mstest

1. First you should create a testlist in testmetadata file(vsmdi)
2. the commandline should be mstest /testmetadata:....vsmdi /testlist:<name>
3. Then use ccnet config to run mstest

Unthread

but that means you have to maintain the .vsmdi file every time you add a new test – Amara

For what its worth, test lists are deprecated as of VS2012. – Overstuffed

Answer 4

i know this thread is quite Old, but its still high on Google so i thought i might help one or two. Anyway, since there is no satisfactory solution for this. I've written an msbuild task for this. details can be found here: http://imistaken.blogspot.com/2010/08/running-all-tests-in-solution.html

Amortize

That's cool and all but how do you make that work in CruiseControl.NET? With EXEC task compiling the tests? – Inhalant

Answer 5

You could enforce some convention on the naming and location of test projects, then you could run MSTest on, say, all *Test.dll below the location of your solution.

As far as I know, there is no way to tell a test project from a 'normal' DLL project based soleley on a solution file. So, an alternative could be to analyze the project files and/or .vsmdi files to find the test projects, but that could be rather tricky.

Sacerdotal

Answer 6

I don't know directly but this is where VSMDI [fx:spits in a corner] can help. In your solution add all the tests to the VSMDI. And then pass the VSMDI to mstest using /testmetadata.

However I would suggest that you follow the conventions above. And use a naming convention and dump that out from the SLN file using say a for loop in the command script

Refract

Answer 7

I would just write a target that calls it the way you want, then whip up a batch file that calls the target that contains all the DLL's to be tested.

Unless you're adding test projects all the time, you'll very rarely need to modify it.

Bono

We add test projects every 4-5 months, putting the project in a "master" solution for "auto building" is already a habbit. i'm trying to minimize adding more steps for people to remember. the scary thing about testing is how easy it is to never run the tests you have written. – Hanafee

Does each non-test project have a single corresponding test project? – Bono

Answer 8

Why not just have msbuild output all the test assemblies to a folder.

Try setting OutputPath,OutputDir,OutDir properties in msbuild to accomplish this.

then have mstest execute against all assemblies in that folder.

Hluchy

the problem with that is the test assemblies are right next to the real assemblies they are testing. sending mstest a command like "*.dll". I'm trying to make this process dumb enough so we always are testing projects that we add. if we add a new test project, which will probably happen every 4-5 months (short enough to forget the steps), i want to process to simply read a solution for test projects. – Hanafee

Do you suffix your test assemblies with .Test ? – Hluchy