CMake exclude tests in subdirectories

Question

I have a c++ project which includes libevent library. Project structure:

.
|_ CMakeLists.txt
|_ Makefile
|_ src
| |_ my_lib.cpp
|_ test
| |_ my_lib_test.cpp
|_ lib
  |_ libevent
    |_ CMakeLists.txt
    |_ ...

When I build and run my tests, libevent tests are also executed. How can I exclude them and run only my own tests?

Blinking

Answer 1

Looking at the available options in libevent's CMakeLists.txt file, it appears that you can disable these pretty easily by setting EVENT__DISABLE_TESTS to ON.

You can either do this in your own CMakeLists.txt before libevent is included:

set(EVENT__DISABLE_TESTS ON)
...
add_subdirectory(lib/libevent)

or when you invoke CMake on the command line:

cmake . -DEVENT__DISABLE_TESTS=ON

Sewerage

What if it's not only libevent next time? Is there a standard CMake way to disable/enable subproject tests? – Blinking

No. You're just lucky in this case that the creators of the libevent CMakeLists.txt have made the effort to provide this option. Lots of other projects do have similar options to avoid building tests/examples/docs/etc. but many do not. For those that don't, you'd have to modify their CMakeLists.txt files (e.g. to remove/skip all instances of the [add_test] command). – Sewerage

Answer 2

There is also a more general way to do it. Add a file named CTestCustom.cmake to your source tree and add a list of tests that you want CTest not to run:

set(CTEST_CUSTOM_TESTS_IGNORE
   test1
   ....
   testN
)

Then copy this file to the build directory where tests are executed:

configure_file(${CMAKE_SOURCE_DIR}/CTestCustom.cmake ${CMAKE_BINARY_DIR})

This will make CTest ignore the listed tests. See this for more info.

Dogtooth

That's not very convenient: I'd like to exclude testing 3rd party libraries, and it is a bit tiresome to enumerate hundreds of tests. – Bdellium

Answer 3

I've found another (better) way to solve the actual question.

TL;DR

Add to your CMakeLists.txt file:

configure_file(${CMAKE_SOURCE_DIR}/CTestCustom.cmake ${CMAKE_BINARY_DIR} @ONLY)

Add do your CTestCustom.cmake file in your source code:

file (STRINGS "@CMAKE_BINARY_DIR@/CTestTestfile.cmake" LINES)

# overwrite the file....
file(WRITE "@CMAKE_BINARY_DIR@/CTestTestfile.cmake" "")

# loop through the lines,
foreach(LINE IN LISTS LINES)
  # remove unwanted parts
  string(REGEX REPLACE ".*directory_to_remove.*" "" STRIPPED "${LINE}")
  # and write the (changed) line ...
  file(APPEND "@CMAKE_BINARY_DIR@/CTestTestfile.cmake" "${STRIPPED}\n")
endforeach()

(See https://mcmap.net/q/745520/-how-to-remove-a-line-of-text-in-a-string-in-cmake-working-around-cmake-39-s-lack-of-line-based-regex-matching for a function form of removing a line)

Explanation:

The excluding per test method has many downsides, including dynamically (via a macro) added tests, wasted time, etc...

This was tested on CMake 3.x: When a CMake project with tests enabled is built, a CTestTestfile.cmake file is generate in the CMAKE_BINARY_DIR. Essentially every time add_subdirectory is called, an equivalent subdirs command is added to the CTestTestfile.cmake file. The CTestTestfile.cmake files will mimic an equivalent directory structure as the CMakeLists.txt files. (So if you need to remove a sub directory in another directory, that's the CTestTestfile.cmake you need to edit).

The CTestCustom.cmake is executed before the CTestTestfile.cmake, so all CTestCustom.cmake needs to do is remove the offending line from the CTestTestfile.cmake file. This could be done with a sed, but in the spirit of cross compatibility, it's all pure CMake now. The CTestCustom.cmake does not have CMAKE_BINARY_DIR set, so we use @ substitution to get the value of CMAKE_BINARY_DIR at CMake generate time, and replace the values that will be in the CTestCustom.cmake copy in the build directory, using @ONLY replacement, to leave the other variables $ alone.

Reclamation

Answer 4

I agree with @Ivan Baidakou.

I'm not sure this is much better, but here's a non-portable hack (won't work on Windows by default) to get it done.

execute_process(COMMAND sh -c "echo 'set(CTEST_CUSTOM_TESTS_IGNORE'; find '${CMAKE_SOURCE_DIR}/relative_paths' -path '*tests/CMakeLists.txt' -exec sed -nE '/add_test.*NAME/{s|.*NAME *([^ ]+).*|\\1|; p}' {} +; echo ')'"
                OUTPUT_FILE ${CMAKE_BINARY_DIR}/CTestCustom.cmake)

Where ${CMAKE_SOURCE_DIR}/relative_paths can be multiple paths for multiple libraries if you want.

Reclamation