How to get CTest results in Hudson / Jenkins

Question

I'm using CTest (part of CMake) for my automated tests.

How do I get CTest results in the Jenkins dashboard ? Or, phrased differently, how do I get CTest to output in JUnit-like XML ?

Capote

Answer 1

In Jenkins, after the CMake part (probably made through the CMake plugin), add the following batch script, or adapt for builds on Linux :

del build_32\JUnitTestResults.xml
pushd build_32\Tests
"C:\Program Files\CMake 2.8\bin\ctest.exe" -T Test -C RelWithDebInfo --output-on-failure
popd
verify >nul
C:\Python27\python.exe external/tool/CTest2JUnit.py build_32/Tests external/tool/CTest2JUnit.xsl > build_32/JUnitTestResults.xml

• build_32 is the Build Directory in the CMake plugin
• Tests is the subdirectory where all my tests live
• -T Test makes CTest output in XML (?!)
• verify >nul resets errorlevel to 0, because CTest returns >0 if any test fails, which Jenkins interprets as "the whole build failed", which we don't want
• The last line converts CTest's XML into a minimal JUnit xml. The Python script and the xslt live in the source directory, you may want to change that.

The python script looks like this (hacked together in 10 min, beware) :

from lxml import etree
import StringIO
import sys

TAGfile = open(sys.argv[1]+"/Testing/TAG", 'r')
dirname = TAGfile.readline().strip()

xmlfile = open(sys.argv[1]+"/Testing/"+dirname+"/Test.xml", 'r')
xslfile = open(sys.argv[2], 'r')

xmlcontent = xmlfile.read()
xslcontent = xslfile.read()

xmldoc = etree.parse(StringIO.StringIO(xmlcontent))
xslt_root = etree.XML(xslcontent)
transform = etree.XSLT(xslt_root)

result_tree = transform(xmldoc)
print(result_tree)

• It needs lxml, direct link
• It takes two arguments, the directory in which the tests live (in the build directory), and a xsl file
• It simply reads the last xml tests results, transforms it with the xsl, and outputs it to stdout
• The "last xml tests" are present in the first line of the Testing/TAG file, hence the additional fopen

The xsl looks like this. It's pretty minimal but gets the job done : [EDIT] see MOnsDaR 's improved version : http://pastebin.com/3mQ2ZQfa

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/>

    <xsl:template match="/Site/Testing">    
        <testsuite>
            <xsl:apply-templates select="Test"/>
        </testsuite>
    </xsl:template>

    <xsl:template match="Test">
        <xsl:variable name="testcasename"><xsl:value-of select= "Name"/></xsl:variable>
        <xsl:variable name="testcaseclassname"><xsl:value-of select= "FullName"/></xsl:variable>
        <testcase name="{$testcasename}" classname="{$testcaseclassname}">
            <xsl:if test="@Status = 'passed'">
            </xsl:if>
            <xsl:if test="@Status = 'failed'">
                <error type="error"><xsl:value-of select="Results/Measurement/Value/text()" /></error>
            </xsl:if>
            <xsl:if test="@Status = 'notrun'">
                <skipped><xsl:value-of select="Results/Measurement/Value/text()" /></skipped>
            </xsl:if>
        </testcase>
    </xsl:template>

</xsl:stylesheet>

Finally, check "Publish JUnit tests results" (or similar, my version is in French) and set the xml path to build_32/JUnitTestResults.xml

Well, that was ugly. But still, hope this helps someone. And improvements are welcome ( running ctest from python maybe ? Using the path of the Python plugin instead of C:... ? )

Capote

Thanks a lot for this solution. I'm currently playing around with it and it basically works perfect for me. – Cluj

I enhanced your XSL-Code to support more features of the CTest XML-Output: pastebin.com/GAj7Fzqd It now adds the execution time of the tests and in case of an error it adds the errortype and errorcode as a message. Please comment if there is some other thing which could be added to the generated JUnit-File. – Cluj

The new version supports the <Property>-Tags of JUnit and additionally stores the properties as system-out (so they could be seen in Jenkins): pastebin.com/3mQ2ZQfa – Cluj

@Cluj : Thanks for the improvements ! Awesome. I'll test it this week when I find some time, and I'll update my answer with your code. – Capote

Updated answer ; I let the link to pastebin, the xsl is a bit too big. Thanks again, works well. – Capote

on Linux you can use xsltproc + shell to achieve the same results: copy XSL file to some location, and then use following command: xsltproc ctest2junix.xsl Testing/head -n 1 < Testing/TAG/Test.xml > JUnitTestResults.xml head -n ... command should be enclosed in backquotes... – Beguile

I further improved xslt file from @Cluj xslt as: CTest2JUnit.xsl It fills testclassname as 'this/Path', tests and failures. – Bushwhacker

I wrote a improved version of that script which is located here – Lilas

Answer 2

This seems to be integrated in jenkins-ci nowadays:

• https://github.com/jenkinsci/xunit-plugin/commits/master/src/main/resources/org/jenkinsci/plugins/xunit/types/ctest-to-junit.xsl

Merited