Doctest failed with zero exit code

Question

In my test code, my doctest fails but the script exits with a zero return value, which causes the CI run to pass, which is not intended.

Is this the correct behavior of doctest module?

My script ends with:

if __name__ == '__main__':
    import doctest
    doctest.testmod()

The output is like:

**********************************************************************
File "test/test.py", line 7, in __main__
Failed example:
    f(1,0)
Expected:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    x
    -----
Got:
    -----
    type: <type 'exceptions.ZeroDivisionError'>
    value: integer division or modulo by zero
    -----
**********************************************************************
1 items had failures:
   1 of   1 in __main__
***Test Failed*** 1 failures.
tux@iPad:~/programming/exception-notifier(fix-travis)(0)$ echo $?
0

Iraq

Can you please post the full stacktrace ? – Subdued

dropping the X from your docstring ? – Mithras

That x is deliberately added to make the test fail. – Iraq

Answer 1

I find using doctest.testmod(raise_on_error=True) will cause an exception to be raised when a test fails, which causes the script exits with a non-zero code.

Python doc here:

Optional argument raise_on_error defaults to false. If true, an exception is raised upon the first failure or unexpected exception in an example. This allows failures to be post-mortem debugged. Default behavior is to continue running examples.

Iraq

Unfortunately using raise_on_error leads to a missing test output, so I switched to sys.exit based solution. – Barnet

@Barnet I'm using verbose flag in combination with raise_on_error to produce output and this allows to figure out which test has failed: doctest.testmod(verbose=True, raise_on_error=True) – Surinam

Answer 2

@fossilet's answer works for properly breaking builds that fail tests, but it raises the exception before doctest is able to write anything to the console. This makes your logs much less useful for identifying the problem.

An alternative is to call

sys.exit(doctest.testmod()[0])

This makes the process exit code equal to the number of tests that failed. Your CI tool should interpret nonzero exit codes as failing builds. But the doctest output will still make it to the console.

Eris

Answer 3

The doctest.testmod function returns a pair with the failure and test counts:

(failure_count, test_count)

So we can exit with an exit code different from zero whenever failure_count is different from / greater than zero:

def foo():
    """
    This amazing function returns 1337

    >>> foo()
    1
    """ 

    return 1337

if __name__ == "__main__":
    import doctest
    import sys
    (failure_count, _) = doctest.testmod()
    if failure_count:
        sys.exit(1)

This works:

$ python example.py
**********************************************************************
File "example.py", line 5, in __main__.foo
Failed example:
    foo()
Expected:
    1
Got:
    1337
**********************************************************************
1 items had failures:
   1 of   1 in __main__.foo
***Test Failed*** 1 failures.

$ echo $?
1

It prints the error and returns an error exit code (1). This is useful if running tests from within a Makefile or in CI: make aborts execution and reports the error; the CI build rightfully fails.

What I like about this solution is that success is silent. If we fix the doctest, this is the output:

$ python example.py

$ echo $?
0

Do not use the count of errors as the exit code

You may be tempted to write:

if __name__ == "__main__":
    import doctest
    import sys
    sys.exit(doctest.testmod()[0]) # XXX: don't!

Don't.

The documentation of sys.exit states: "Most systems require [the exit code] to be in the range 0–127, and produce undefined results otherwise."

On Linux, exit codes are modulo 256. So if one has exactly 256 doctests wrong, one would get a success (0) exit code! That would be true for all multiples of 256: 512, 768, 1024, ... On other systems you end up relying on undefined behaviour: maybe you'll always get a success (0) when you have over a hundred errors, which is the opposite of what you would have wanted.

Alternative: using raise_on_error=True

We could use the raise_on_error argument to doctest.testmod so that it throws an error upon a failing test:

    doctest.testmod(raise_on_error=True)

However, this prints a traceback which is not very useful:

$ python example.py
Traceback (most recent call last):
  File "example.py", line 13, in <module>
    doctest.testmod(raise_on_error=True)
  File "/usr/lib/python3.12/doctest.py", line 1998, in testmod
    runner.run(test)
  File "/usr/lib/python3.12/doctest.py", line 1886, in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/doctest.py", line 1525, in run
    return self.__run(test, compileflags, out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/doctest.py", line 1426, in __run
    self.report_failure(out, test, example, got)
  File "/usr/lib/python3.12/doctest.py", line 1895, in report_failure
    raise DocTestFailure(test, example, got)
doctest.DocTestFailure: <DocTest __main__.foo from example.py:1 (1 example)>

One could activate verboseness (verbose=True) to get a summary before the traceback. However the useless traceback still pollutes the output and success stops being silent. So I personally don't like this alternative very much.

As of 2024, there is no other option in doctest.testmod to directly exit with an error code. Something like doctest.testmod(exit_on_error=1) would be nice...

Luanaluanda