Running only changed or failed tests with CMake/CTest?

Question

I work on a large code base that has close to 400 test executables, with run times varying between 0.001 second and 1800 seconds. When some bit of code changes CMake will rebuild intelligently only the targets that have changed, many times taking shorter than the actual test run will take.

The only way I know around this is to manually filter on tests you know you want to run. My intuition says that I would want to re-run any test suite that does not have a successful run stored - either because it failed, or because it was recompiled.

Is this possible? If so, how?

Ene

If your test generates a file on success, make it depend of unitTest executable should do most of the job. – Caoutchouc

@Caoutchouc But how do I tie that to CTest? I'm still a little vague on how CTest and CMake interact... Just putting it in CMake as a custom target would work, but lose the CTest functionality completely. – Ene

You said that you have "400 test executables" (by this I understand 400 main) so you selectively run from these executable. – Caoutchouc

Using CTest I can only run the executables - either all, or all that failed last time. I cannot get it to rerun all changed successful executables. I could do that with a custom command in cmake or something like that, or by modifying the arguments, but the whole point is to not do it manually. – Ene

ctest command accepts several parameters, which affects on set of tests to run. E.g. ,"-R" - filter tests by name, "-L" - filter tests by label. Probably, using dashboard-related options, you may also choose tests to run. As for generating value of these options according to changed executables, you may write program or script. Another way for run only selected test executables is to wrap every executable into script, which runs executable only when specific condition is applied. – Argosy

@Argosy If you make that into a reply I'll tag it as answer. Didn't think of making it into a wrapper for the test executable, but it makes a lot of sense - essentially ccache-wrapping the test. – Ene

Answer 1

ctest command accepts several parameters, which affects on set of tests to run. E.g. ,"-R" - filter tests by name, "-L" - filter tests by label. Probably, using dashboard-related options, you may also choose tests to run.

As for generating values for these options according to changed executables, you may write program or script, which checks modification time of executables and/or parses last log file for find failed tests.

Another way for run only changed executables is to wrap tests into additional script. This script will run executable only if some condition is saticfied.

For Linux wrapper script could be implemented as follows:

test_wrapper.sh:

# test_wrapper.sh <test_name> <executable> <params..>
# Run executable, given as second argument, with parameters, given as futher arguments.
#
# If environment variable `LAST_LOG_FILE` is set,
# checks that this file is older than the executable.
#
# If environment variable LAST_LOG_FAILED_FILE is set,
# check that testname is listed in this file.
#
# Test executable is run only if one of these checks succeed, or if none of checks is performed.

check_succeed=
check_performed=
if [ -n $LAST_LOG_FILE ]; then
    check_performed=1
    executable=$2
    if [ ! ( -e "$LAST_LOG_FILE" ) ]; then
        check_succeed=1 # Log file is absent
    elif [ "$LAST_LOG_FILE" -ot "$executable" ]; then
        check_succeed=1 # Log file is older than executable
    fi
fi

if [ -n "$LAST_LOG_FAILED_FILE" ]; then
    check_performed=1
    testname=$1
    if [ ! ( -e "$LAST_LOG_FAILED_FILE" ) ]; then
        # No failed tests at all
    elif grep ":${testname}\$" "$LAST_LOG_FAILED_FILE" > /dev/null; then
        check_succeed=1 # Test has been failed previously
    fi
fi

if [ -n "$check_performed" -a -z "$check_succeed" ]; then
    echo "Needn't to run test."
    exit 0
fi

shift 1 # remove `testname` argument
eval "$*"

CMake macro for add wrapped test:

CMakeLists.txt:

# Similar to add_test(), but test is executed with our wrapper.
function(add_wrapped_test name command)
    if(name STREQUAL "NAME")
        # Complex add_test() command flow: NAME <name> COMMAND <command> ...
        set(other_params ${ARGN})
        list(REMOVE_AT other_params 0) # COMMAND keyword
        # Actual `command` argument
        list(GET other_params 0 real_command)
        list(REMOVE_AT other_params 0)
        # If `real_command` is a target, need to translate it to path to executable.
        if(TARGET real_command)
            # Generator expression is perfectly OK here.
            set(real_command "$<TARGET_FILE:${real_command}")
        endif()
        # `command` is actually value of 'NAME' parameter
        add_test("NAME" ${command} "COMMAND" /bin/sh <...>/test_wrapper.sh
            ${command} ${real_command} ${other_params}
        )
    else() # Simple add_test() command flow
        add_test(${name} /bin/sh <...>/test_wrapper.sh
            ${name} ${command} ${ARGN}
        )
    endif()
endfunction(add_wrapped_test)

When you want to run only those tests, which executables have been changed since last run or which has been failed last time, use

LAST_LOG_FILE=<build-dir>/Testing/Temporary/LastTest.log \
LAST_FAILED_LOG_FILE=<build-dir>/Testing/Temporary/LastTestsFailed.log \
ctest

All other tests will be automatically passed.

Argosy

Thanks for this. The *.sh may also be any script language. Once i use this i may port it to a CMake script which is multiplatform. CMake is obviously installed ;) – Brainstorming