How to get passed and fail test case count in xunit using cake(c# make) script

Question

I try to use the cake script for running the test cases written in Xunit using the cake script , I need to know the number of passed and failed test cases count.

#tool "nuget:?package=xunit.runner.console"
var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies);

Reference : http://www.cakebuild.net/dsl/xunit-v2

Can anyone please suggest how to get the number of passed and failed test cases?

Syrupy

Are you referring to the test report or are you looking to get those values to be used for something else? – Faydra

@Faydra XUnit2(testAssemblies); This line will run the test cases in the mentioned DLL – Syrupy

@Faydra I want to get the test case run summary like failed count ,passes test case count , or Even simply it is passed or failed in code level – Syrupy

XUnit2Aliases​.XUnit2(IEnumerable<string>, ​XUnit2Settings) allows you to run the tests with settings. One of which is whether to create a test report which would include those values. – Faydra

@Faydra Thank you, I will try – Syrupy

Answer 1

You'll have to use XUnit2Aliases​.XUnit2(IEnumerable < FilePath >, ​XUnit2Settings) + XmlPeekAliases for reading the XUnit output.

var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
XUnit2(testAssemblies,
     new XUnit2Settings {
        Parallelism = ParallelismOption.All,
        HtmlReport = false,
        NoAppDomain = true,
        XmlReport = true,
        OutputDirectory = "./build"
    });

The xml format is:(XUnit documentation, the example source, more information in Reflex)

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="1" errors="1" failures="0" skip="0">
    <testcase classname="path_to_test_suite.TestSomething"
              name="test_it" time="0">
        <error type="exceptions.TypeError" message="oops, wrong type">
        Traceback (most recent call last):
        ...
        TypeError: oops, wrong type
        </error>
    </testcase>
</testsuite>

Then the following snippet should bring you the information:

var file = File("./build/report-err.xml");
var failuresCount = XmlPeek(file, "/testsuite/@failures");
var testsCount = XmlPeek(file, "/testsuite/@tests");
var errorsCount = XmlPeek(file, "/testsuite/@errors");
var skipCount = XmlPeek(file, "/testsuite/@skip");

Isolecithal

Answer 2

Like most test runners, XUnit returns the number of failed tests in the return code from the console runner. Out of the box, Cake throws an exception, and therefore fails the build, when the return code of a tool is NOT zero.

This can be seen in the XUnit Runner Tests here:

https://github.com/cake-build/cake/blob/08907d1a5d97b66f58c01ae82506280882dcfacc/src/Cake.Common.Tests/Unit/Tools/XUnit/XUnitRunnerTests.cs#L145

Therefore, in order to know whether:

simply it is passed or failed in code level

This is known implicitly by whether or not the build succeeded or not. I typically use a strategy similar to this:

Task("Tests")
.Does(() =>
{
    var testAssemblies = GetFiles("./src/**/bin/Release/*.Tests.dll");
    XUnit2(testAssemblies,
        new XUnit2Settings {
            Parallelism = ParallelismOption.All,
            HtmlReport = false,
            NoAppDomain = true,
            XmlReport = true,
            OutputDirectory = "./build"
    });
})
.ReportError(exception =>
{
    Information("Some Unit Tests failed...");
    ReportUnit("./build/report-err.xml", "./build/report-err.html");
});

This is making use of the Exception Handling capabilities in Cake:

http://cakebuild.net/docs/fundamentals/error-handling

To take action when an error occurs. On top of that, I am then using the ReportUnit alias to convert the XML Report into a human readable HTML Report.

Hendiadys