Using CMake, how do I get verbose output from CTest?

Question

I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:

ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)

But the build output in Visual Studio only shows the result of running CTest:

      Start 1: UnitTests
  1/1 Test #1: UnitTests ................***Failed    0.05 sec

  0% tests passed, 1 tests failed out of 1

This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose I get the output from a Boost unit test which tells what actually failed:

1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed    0.00 sec

So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose at all times? Is there a better way to use Boost unit tests with CMake/CTest?

Wanitawanneeickel

Does this answer your question? Replace ctest command with "ctest --output-on-failure" permanently for a specific project in CMakeLists.txt – Wagonage

Answer 1

You can use the ctest --output-on-failure option, or set the environment variable CTEST_OUTPUT_ON_FAILURE, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:

env CTEST_OUTPUT_ON_FAILURE=1 make check

This Stack Overflow question and answer shows how to set environment variables in Visual Studio.

Devinna

Doesn't work for me (ctest version 2.8.12.1). I tried SET(CTEST_OUTPUT_ON_FAILURE TRUE) and SET(CTEST_OUTPUT_ON_FAILURE ON), but it has no effect. Other reports in the web corroborate that this is broken. – Hairston

@JoachimWuttke set(CTEST_OUTPUT_ON_FAILURE TRUE) doesn't set an environment variable. Try this on the command line: CTEST_OUTPUT_ON_FAILURE=TRUE make test. – Duky

make CTEST_OUTPUT_ON_FAILURE=1 test is shorter and nicer IMO. – Humor

On windows batch file, how to use CTEST_OUTPUT_ON_FAILURE=1 while calling -- msbuild /toolsversion:15.0 /p:Configuration=Release /p:Platform=x64 TESTS.vcxproj – Caylacaylor

Same as --output-on-failure according to the manual: man 1 ctest. – Fescennine

@Devinna What's make check? And what I often see is make test other than make check. – Muddleheaded

Answer 2

You could call ctest directly, after cmaking and making your project.

ctest --verbose

Chandos

They mentioned that option in the original post but said they were looking to avoid having to run it manually. – Encumber

Answer 3

There is a very simple solution (which for some reason is difficult to find via Google Search):

ctest --output-on-failure

If you use CMake with Visual Studio's open folder function you can add the

"ctestCommandArgs": "--output-on-failure"

setting to your build configuration.

Smaze

What if I want to output on success? – Recreant

Answer 4

1. You can check the Testing/Temporary subfolder. It is automatically created after running make test. This folder contains two files: LastTest.log and LastTestsFailed.log. LastTest.log contains desired output for run tests. LastTestFailed.log contains names of failed tests. So you can check them manually after executing make test.

2. The second way is to get ctest to show you the content of log files after running tests:

   1. place in build dir (from which you run make test) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents

      CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")

Instead of cat you may use whatever Windows cmd command that does similar things.

2. run make test again and get profit!

additional info about customizing ctest you can find here. Just step to "Customizing cmake" section. Good luck!

Endear

Okay, thanks. I was hoping there would be a way to modify the flags that are inserted into the project/makefiles for ctest, but I didn't find anything and your answer seems to confirm that. The file name info is helpful! – Wanitawanneeickel

somewhere around ctest 2.8.10 they have broken using external commands with arguments in CTEST_CUSTOM_POST_TEST see github.com/openscad/openscad/issues/260 – Archaimbaud

@don: maybe they aren't running enough tests on ctest ;-) – Hairston

That CMake issue with CTEST_CUSTOM_POST_TEST is fixed in 2.8.12. – Admix

CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log") does work. But it displays logs from previous CTest run :( – Waly

Answer 5

This makes test output more verbose:

make test ARGS="-V"

Catamenia

Nice, exactly what I was looking for. This made sure that my output printed with print_message was shown even when all tests SUCCEEDED - otherwise I only saw those prints on failure. – Rush

IMO, this is the only correct answer. The "output on failure" answers do not qualify, as one may need to produce reports/results, in case of successful execution as well. – Recreant

Answer 6

I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:

add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)

Odeen

Answer 7

make check CTEST_OUTPUT_ON_FAILURE=TRUE

Brick

All the down-votes are rather surprising: this is almost equivalent to the accepted answer, but shorter and nicer. Also works in the project I tested. – Brick

make CTEST_OUTPUT_ON_FAILURE=1 test – Sanburn

Answer 8

My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND} variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1 argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>. To help clarify what is going on, I start with three cmake -E echo commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target.

add_custom_target(check 
        ${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
        COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
        COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
        COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
            ${CMAKE_CTEST_COMMAND} -C $<CONFIG>
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
    DEPENDS ALL_BUILD
    )

This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E portable command line tool mode. I also add a dependency on ALL_BUILD so that all projects will be built before invoking the check target. (On Linux builds, one may need to replace ALL_BUILD with ALL; I have not tested this on Linux yet.)

Cypress

Answer 9

There's now a CMake variable that allows you to modify the behaviour of make test. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test.

So adding this to your CMake file (must be above enable_testing()):

set(CMAKE_CTEST_ARGUMENTS "--verbose")

Means CTest will always run verbose. Or for just the output of the failed tests, use:

set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")

Edit: As suggested by RobLoach, since it's a list of arguments, you'll want to append to the list instead.

list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")

Eggbeater

Since it's a list of arguments, you'll want to append to the list... list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure") – Underpart

Answer 10

For people using Visual Studio, here another variation (hack) on the theme:

cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS

Westlund

Answer 11

ctest -VV or ctest --extra-verbose

From documentation:

Enable more verbose output from tests.

Test output is normally suppressed and only summary information is displayed. This option will show even more test output.

Grochow

Answer 12

to show the result with XML file you have to execute the test with the following command

~$ ctest -T Test

and we found the result in the Testing/1234123432/test.xml and other files are generated too in Testing Folder

Colquitt

With -T there is some summary information in the XML, but it only includes the top tests, not individual test cases. Is there a way to also display those? – Recreant