test for non-zero exit status using ctest/cmake

Question

The application of interest is a compiler which returns a non-zero exit code when it encounters an error in the source. The unit tests for the compiler are composed of small snippets which intentionally triggers errors.

The function used to add a test is:

function(add_compiler_test test_name options)
  add_test(NAME ${test_name}
    COMMAND $<TARGET_FILE:pawncc> ${DEFAULT_COMPILER_OPTIONS} ${options})
  set_tests_properties(${test_name} PROPERTIES
    ENVIRONMENT PATH=$<TARGET_FILE_DIR:pawnc>)
endfunction()

This causes the test to fail when the program returns a non-zero exit code despite that being the correct behaviour.

How can the exit status of the program be tested?

Ammoniate

Answer 1

If you want the test(s) to be reported as SUCCESS when it returns non-zero exit status and FAILED otherwise, set WILL_FAIL property for the test(s):

set_tests_properties(<test1> <test2> ... PROPERTIES WILL_FAIL TRUE)

Nickles

How do I distinguish between the program crashing and program intentionally returning a non-zero exit code? I need to check the exit status. – Ammoniate

Crashing will be reported as FAILED in any case. See also that question and my answer for it. – Nickles

The regular expression tests (using PASS_REGULAR_EXPRESSION) are succeeding despite actually failing because of setting WILL_FAIL to TRUE. How do I go about resolving this? I need to force an output check but allow for a non-zero exit code. – Ammoniate

Same here. Looking for a way to check against a specific non-zero return value and specific output. @Yashas, have you found a solution to this? I'm also testing a compiler btw :D – Hege

@sebkraemer: For complex interpretation of the executable result (in your case, the compiler result) you may create a wrapper around calling that executable. See that my answer for the related question. – Nickles

@Tsyvarev, I figured I have to go one turn of indirection, a script or similar. To gain full flexibility (return code and output), I'm already considering pulling in a test framework and do the assertions as unit tests. I might need one later on anyway. Still thanks for the neat little program, I'll keep it in mind. – Hege