CMake & CTest : make test doesn't build tests

Question

I'm trying CTest in CMake in order to automatically run some of my tests using make test target. The problem is CMake does not "understand" that the test I'm willing to run has to be built since it is part of the project.

So I'm looking for a way to explicitly specify this dependency.

Caleb

Answer 1

It is arguably a bug in CMake (previously tracked here) that this doesn't work out of the box. A workaround is to do the following:

add_test(TestName ExeName)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
                  DEPENDS ExeName)

Then you can run make check and it will compile and run the test. If you have several tests, then you would have to use DEPENDS exe1 exe2 exe3 ... in the above line.

Contessacontest

so I guess that the "make test" target will remain unused as it seems you have to choose a different target name ine the add_custom_target command? – Caleb

Yep. The only difference between "make test" and "make check" is the former shows "Running tests..." first and doesn't check any build dependencies. – Contessacontest

@rq - but how can I do this with multiple projects (when one CMakeLists.txt is sub-project of other) so each one would define check target and they may collide – Presumption

@Presumption - in that case you're probably better of just using the equivalent "make all test". In fact, this is what I do anyway. – Contessacontest

Actually, some consider it a feature (not a bug) of cmake that you can run "make test" and just run the tests as they are without doing any re-builds first... – Suzannesuzerain

I did this and now it claims to run tests but actually doesn't. It says 1 out of 1 tests are successful but it does not run any tests! When I deliberately fail a test it does not see it. When I manually execute test/test_myapp the CUnit tests run properly. WTH? – Psaltery

This doesn't work in IDE builds because "-C Debug" or similar needs to be passed to ctest. CMAKE_CFG_INTDIR Doesn't work because the test command isn't known at configure time. – Parrie

Answer 2

There is actually a way to use make test. You need to define the build of the test executable as one of the tests and then add dependencies between the tests. That is:

ADD_TEST(ctest_build_test_code
         "${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target test_code)
ADD_TEST(ctest_run_test_code test_code)
SET_TESTS_PROPERTIES(ctest_run_test_code
                     PROPERTIES DEPENDS ctest_build_test_code)

Rain

This is the only one that scales up and doesn't force you to build the "make all" targets just in order to run the test. A possible downside: details of build errors on the binaries only show up in the generated LastTest.log file and not on stdout/stderr – Papiamento

Good answer! You should add the config to the build target though. Otherwise it's not possible to run the tests in all configurations. add_test( NAME "${ARGV0}_BUILD" COMMAND "${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target ${target} "--config" "$<CONFIG>" ) – Bybidder

This clogs up the test reporter with a bunch of phony tests. – Deflected

If you are using CMake >= 3.7, the recommended approach is to use fixtures. See my answer below. – Yahoo

Answer 3

If you are using CMake >= 3.7, then the recommended approach is to use fixtures:

add_executable(test test.cpp)
add_test(test_build
  "${CMAKE_COMMAND}"
  --build "${CMAKE_BINARY_DIR}"
  --config "$<CONFIG>"
  --target test
)
set_tests_properties(test_build PROPERTIES FIXTURES_SETUP    test_fixture)
add_test(test test)
set_tests_properties(test       PROPERTIES FIXTURES_REQUIRED test_fixture)

This does the following:

• Adds a test executable target built from test.cpp
• Adds a test_build "test" that runs Cmake to build target test
• Marks the test_build test to be a setup task of fixture test_fixture
• Add a test test that just runs the test executable
• Marks the test test to need fixture test_fixture.

So, every time test test is to be run, it first runs test test_build, which builds the necessary executable.

Yahoo

If $<CONFIG> is not set the --target will become the argument for the --config. – Imperious

I believe $<CONFIG> is always non-empty. It is a generator expression for the configuration name: cmake.org/cmake/help/latest/manual/… I'll edit the answer to wrap it in quotes anyway just because it makes no difference. – Yahoo

How do you run cmake? I do this way: mkdir build; cd build; cmake ..; make. And it looks like there is no any defaults and all related variables are empty, until CMAKE_BUILD_TYPE is set manually. (currently on Debian 10, didn't check other platforms) – Imperious

I am not completely sure: can you run several make processes over the same build directory concurrently? This situation may happen, because tests can be executed concurrently by ctest – Magree

Answer 4

I use a variant of richq's answer. In the top-level CMakeLists.txt, I add a custom target, build_and_test, for building and running all tests:

find_package(GTest)
if (GTEST_FOUND)
    enable_testing()
    add_custom_target(build_and_test ${CMAKE_CTEST_COMMAND} -V)
    add_subdirectory(test)
endif()

In the various sub-project CMakeLists.txt files under test/, I add each test executable as a dependency of build_and_test:

include_directories(${CMAKE_SOURCE_DIR}/src/proj1)
include_directories(${GTEST_INCLUDE_DIRS})
add_executable(proj1_test proj1_test.cpp)
target_link_libraries(proj1_test ${GTEST_BOTH_LIBRARIES} pthread)
add_test(proj1_test proj1_test)
add_dependencies(build_and_test proj1_test)

With this approach, I just need to make build_and_test instead of make test (or make all test), and it has the benefit of only building test code (and its dependencies). It's a shame I can't use the target name test. In my case, it's not so bad because I have a top-level script that does out-of-tree debug and release (and cross-compiled) builds by calling cmake and then make, and it translates test into build_and_test.

Obviously, the GTest stuff isn't required. I just happen to use/like Google Test, and wanted to share a complete example of using it with CMake/CTest. IMHO, this approach also has the benefit of allowing me to use ctest -V, which shows the Google Test output while the tests run:

1: Running main() from gtest_main.cc
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from proj1
1: [ RUN      ] proj1.dummy
1: [       OK ] proj1.dummy (0 ms)
1: [----------] 1 test from proj1 (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [  PASSED  ] 1 test.
1/2 Test #1: proj1_test .......................   Passed    0.03 sec

Thermoelectricity

In this example, is there a way to get make test to do what ctest -V does instead of ctest? The ctest output looks very incomplete and just says that there is a single test. – Bullring

Answer 5

If you are trying to emulate make check, you may find this wiki entry usefull :

http://www.cmake.org/Wiki/CMakeEmulateMakeCheck

I have just checked that is does what it says with success (CMake 2.8.10).

Orlandoorlanta

This will build all executables when running make check. For tests with dominating compilation times, this makes ctest -R useless. – Larimore

Answer 6

Save yourself the headache:

make all test

Works out of the box for me and will build dependencies before running the test. Given how simple this is, it almost makes the native make test functionality convenient because it gives you the option of running the last compiling tests even if your code is broken.

Gebelein

Does not work with CDash. You have to call make all && ctest and then the building is not part of the uploaded testing. So build warnings or errors are not visible. – Larimore

Also does not work well if you want a parallel build, since the two will run in parallel: you need make -j4 all && make test. And it's also flaky using a non-Make build tool. – Chaplain

Answer 7

For CMake 3.10 or later, another option is to use the TEST_INCLUDE_FILES directory property to set up a script that triggers a build before a test is run. In your outermost CMakeLists.txt add the following code:

set_property(DIRECTORY APPEND
    PROPERTY TEST_INCLUDE_FILES "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake"
   "execute_process(COMMAND \"${CMAKE_COMMAND}\""
   " --build \"${CMAKE_BINARY_DIR}\""
   " --config \"\$ENV{CMAKE_CONFIG_TYPE}\")")

The actual test configuration is passed through to the build via the environment variable CMAKE_CONFIG_TYPE. Optionally you can add a --target option to only build targets required by the test.

Unfounded

Answer 8

This is what I hammered out and have been using:

set(${PROJECT_NAME}_TESTS a b c)

enable_testing()
add_custom_target(all_tests)
foreach(test ${${PROJECT_NAME}_TESTS})
        add_executable(${test} EXCLUDE_FROM_ALL ${test}.cc)
        add_test(NAME ${test} COMMAND $<TARGET_FILE:${test}>)
        add_dependencies(all_tests ${test})
endforeach(test)

build_command(CTEST_CUSTOM_PRE_TEST TARGET all_tests)
string(CONFIGURE \"@CTEST_CUSTOM_PRE_TEST@\" CTEST_CUSTOM_PRE_TEST_QUOTED ESCAPE_QUOTES)
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake" "set(CTEST_CUSTOM_PRE_TEST ${CTEST_CUSTOM_PRE_TEST_QUOTED})" "\n")

YMMV

Precatory

Answer 9

Derrick's answer, simplified and commented:

# It is impossible to make target "test" depend on "all":
# https://gitlab.kitware.com/cmake/cmake/-/issues/8774
# Set a magic variable in a magic file that tells ctest
# to invoke the generator once before running the tests:
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake"
    "set(CTEST_CUSTOM_PRE_TEST ${CMAKE_MAKE_PROGRAM})\n"
)

It is not perfectly correct, as it does not solve the concurrency problem of running ninja all test, in case anyone does that. On the contrary, because now, you have two ninja processes.

(Ftr, I also shared this solution here.)

Dishevel

Answer 10

All above answers are perfect. But actually CMake use CTest as its testing tools, so the standard method (I think it is) to do the mission is:

enable_testing ()
add_test (TestName TestCommand)
add_test (TestName2 AnotherTestCommand)

Then run cmake and make to build the targets. After that, you can either run make test, or just run

ctest

you will get the result. This is tested under CMake 2.8 .

Check details at: http://cmake.org/Wiki/CMake/Testing_With_CTest#Simple_Testing

Gipon

Downvoted because sometimes you only want to build the targets required for the tests actually being run. – Papiamento

This answer seems to misunderstand the question: The OP is already doing exactly as this answer recommends: Using CTest, enable_testing(), add_test(), etc. The problem is that he has to manually issue the build command prior to running tests. He wants the make test target to automatically build the test executables as necessary. – Refuse

Answer 11

All the answers are good, but they imply a breach of tradition to run a test by command make test. I've done this trick:

add_test(NAME <mytest>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND sh -c "make <mytarget>; $<TARGET_FILE:<mytarget>>")

This means that the test consists of building (optionally) and running of executable target.

Menial

:-D Rule #1: Don't use system without sh. Do you know such system? – Menial

Yes, Windows is one of these. – Sayre

This is also hardcoded to make and loses CMake's feature of generating scripts for other build tools. – Chaplain