CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)

Question

I want ctest to show me the failed tests output by default. That is, I want to run:

$ make all test

and see any output of failed tests without having to cat Testing/Temporary/LastTest.log.

It appears that there are two ways of doing this:

(1) Setting the CTEST_OUTPUT_ON_FAILURE environmental variable:

 $ CTEST_OUTPUT_ON_FAILURE=1 make all test
 $ # or CTEST_OUTPUT_ON_FAILURE=1 ctest

(2) Specifying the --output-on-failure flag to the ctest invocation:

 $ ctest --output-on-failure

Is there a way to write a CMakeLists.txt file such that ctests dumps failed tests output by default on a normal "make all test" invocation WITHOUT resorting to exporting the environmental variable globally in the session or resorting to a custom target like make check (as described here)?

I am aware of the SET_TESTS_PROPERTIES() command, but trying it out like this:

CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
PROJECT(testenv CXX)
ENABLE_TESTING()
ADD_EXECUTABLE(hello hello.cpp)
ADD_TEST(testhello hello)

# Following sets the environment variable for the shell in which the test
# progoram 'hello' is run, but not the shell in which ctest is run
SET_TESTS_PROPERTIES(testhello
    PROPERTIES ENVIRONMENT "CTEST_OUTPUT_ON_FAILURE=1")

and experimenting shows that the environmental variable is set in the shell that the test program is executed in, but not in the shell that ctest is executed in.

Schramke

Does this answer your question? Replace ctest command with "ctest --output-on-failure" permanently for a specific project in CMakeLists.txt – Southernmost

Answer 1

The built-in test target cannot be modified, but you can add a custom check target which invokes ctest with the --output-on-failure switch in the following way:

if (CMAKE_CONFIGURATION_TYPES)
    add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} 
        --force-new-ctest-process --output-on-failure 
        --build-config "$<CONFIGURATION>")
else()
    add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} 
        --force-new-ctest-process --output-on-failure)
endif()

The custom target has to be set up differently for single build type and multi-configuration builds. In the latter case, the active build configuration has to be passed on to the ctest invocation using the --build-config flag. The --force-new-ctest-process is used by the built-in test target by default.

Nitza

Thanks a heap, exactly what i was looking for! Any idea why it isn't possible to modify the test target? – Gooding

The test target is one of CMake's global default targets which cannot be modified. The set up of the test command in hard-coded in the CMake source. – Nitza

truly sad that it is not possible to just change the behavior of the test command by setting the option by default or setting an environment variable... – Alcine

is there still no better option? this defeats the purpose of a simple commands like GTEST_ADD_TESTS. – Randallrandan

At least with the Ninja generator, this doesn't print the test summaries during testing, only after all tests have run. Adding USES_TERMINAL fixes this. – Hegira

Note: I don't think the if is required. The first case will always work AFAIK – Abigailabigale

Answer 2

There is now the list variable CMAKE_CTEST_ARGUMENTS in which you can set ctest arguments.

https://cmake.org/cmake/help/latest/variable/CMAKE_CTEST_ARGUMENTS.html

Repute

Is there some way to debug this? Because I've tried setting it in a variety of ways to no avail. I'm currently setting it in my test directory's CMakeLists.txt, and I've tried moving it around to various other positions within that, but there's no sign it does anything. I'm on version 3.20.2. – Crosslet

Ah, it needs to be set before any call to enable_testing(), just in case that's not obvious to people. – Crosslet