Producing CTest results in Jenkins (xUnit >= 1.58)

Question

It seems like this should be easy to integrate CMake+CTest in jenkins. The cmakebuilder plugin is extremely easy to configure (just set the source tree and the build tree, done!). However I fail to understand how to call the CTest steps.

According to the main xUnit page, since version 1.58 the XML output from CTest is supported, see bug report.

That's about all the documentation I could find. When I search on google or on stackoverflow, I can only find very old documentation requiring manual steps.

I would like to know how to setup a recent jenkins (1.532.1) with xUnit (1.81). Should I create a 'Add build-step' ? Should I create a 'post-build action' ? What do I need to fill in to get CTest to run and to produce proper XML files, so that jenkins can integrate them ?

Babin

Answer 1

Here is a small example that demonstrates how to have xUnit pick up a CTest generated XML test result file. The example consists of a single C++ source file main.cpp

#include <cstdlib>
int main() {
    std::exit(-1);
}

and an accompanying CMakeLists.txt:

cmake_minimum_required(VERSION 2.8)
project(JenkinsExample)
enable_testing()
add_executable(main main.cpp)
add_test(test1 main)
add_test(test2 main)
set_tests_properties(test2 PROPERTIES WILL_FAIL ON)

The CMake list file generates an executable main and runs this executable from two CMake tests, where for demonstration purposes one will always fail and the other one will always succeed.

Using Jenkins, set up a new job and a add new cmakebuilder build step. Point the CMake source directory to the folder that contains the example project on your local machine. The CMake build directory should be set to build. This will make Jenkins create a CMake build folder in the job's workspace directory. It's a good idea to set the Clean Build flag to make the job always start with a clean state.

Then, assuming you are running Unix, add an Execute shell job step and enter the following shell script in the Command box:

cd build
/usr/local/bin/ctest --no-compress-output -T Test || /usr/bin/true

Running ctest with the option -T Test will make CTest generate an XML output file in a sub-folder Testing inside the build folder, which can be picked up by the xUnit plug-in in a post-build action then. The || /usr/bin/true is necessary to prevent Jenkins from aborting the build prematurely (without running the xUnit plug-in) if some tests fail.

If you are using Windows, set up a similar Execute Windows batch command job step instead:

cd build
"C:\Program Files (x86)\CMake 2.8\bin\ctest.exe" --no-compress-output -T Test || verify > NUL

Finally the xUnit plug-in must be configured in the following way:

Add a Publish xUnit test result report post-build action and then use the plug-in's Add button to create a CTest-Version test result report. In the CTest-Version (default) Pattern enter the file pattern build/Testing/**/Test.xml.

Dugout

Two comments, 1. you are missing --no-compress-output from CTest execution. Without this flag you'll end up with issues.jenkins-ci.org/browse/JENKINS-21737 2. To have things simplified, I did register a bug for CTest to control the returned value: cmake.org/Bug/view.php?id=14749 – Babin

@Babin Thanks for the --no-compress-output hint. I have updated the answer. – Dugout

This answer is why I like Stackoverflow so much. Thanks a lot, great help! – Howlett

On Ubuntu 14.04, the shell command should be /usr/bin/ctest --no-compress-output -T Test || /bin/true due to different locations of the binaries – Poynter

This is the method we're using in our builds. However, we're having a problem where if one of the unit test executables crashes, it never produces any XML to be picked up by xUnit, so our builds show up as passing when in reality we've got a test dumping core. How can capture this failure condition? – Lippi

Have you tried setting xUnit's "Fail the build if test results were not updated this run" flag? – Dugout

You can just use ctest --no-compress-output -T Test || true. There is no need to include the full paths. – Voodoo

Answer 2

For those wanting to parse CTest output in a Jenkins Declarative Pipeline, you can now do this quite easily using the xUnit plugin, since it can parse CTest output directly.

Add a stage in your Jenkinsfile to run ctest in the build directory, and add a post stage to process the output with xUnit. Here's a skeleton example:

pipeline {
  agent any
  stages {
    stage('Configure') {
      steps {
        dir('build') {
          sh 'cmake ../'
        }
      }
    }
    stage('Build') {
      steps {
        dir('build') {
          sh 'cmake --build .'
        }
      }
    }  
    stage('Test') {
      steps {
        dir('build') {
          sh 'ctest -T test --no-compress-output'
        }
      }
    }
  }
  post {
    always {
      // Archive the CTest xml output
      archiveArtifacts (
        artifacts: 'build/Testing/**/*.xml',
        fingerprint: true
      )

      // Process the CTest xml output with the xUnit plugin
      xunit (
        testTimeMargin: '3000',
        thresholdMode: 1,
        thresholds: [
          skipped(failureThreshold: '0'),
          failed(failureThreshold: '0')
        ],
      tools: [CTest(
          pattern: 'build/Testing/**/*.xml',
          deleteOutputFiles: true,
          failIfNotNew: false,
          skipNoTestFiles: true,
          stopProcessingIfError: true
        )]
      )

      // Clear the source and build dirs before next run
      deleteDir()
    }
  }
}

For examples of how to configure the xUnit plugin in a Declarative Pipeline, the Jenkins Snippets Generator is a very helpful resource.

Catch

Answer 3

Since CMake 3.21.0 there is also a new option in CTest:

$ man ctest
[...]
  --output-junit <file>        = Output test results to JUnit XML file.

Reference:

• ctest: Add support for generating xUnit/JUnit XML output

Babin