How can I limit the maximum running time for a unit test?

Question

I am currently running some unit tests that might either take a long time before failing or run indefinitely. In a successful test run they will always complete within a certain amount of time.

Is it possible to create a pytest unit test that will fail if it does not complete within a certain amount of time?

Farica

Answer 1

you can install the pytest-timeout plugin and then mark your test functions with a timeout in seconds.

@pytest.mark.timeout(300)
def test_foo():
   pass

Look at the plugin download and usage instructions at https://pypi.python.org/pypi/pytest-timeout

Resist

is this doable without installing an extra package? – Imray

@Imray if you really need to you could vendor in this dependency or write a similar decorator yourself. – Farica

This doesn't seem to work in my case: but admittedly I am using pytest-qt and the cause of the "hang" is a QMessageBox calling question, which causes the test to just stop mid-flow if there is no user input. Maybe pytest-qt has something specific... ? – Wildee

pytest-timeout: 5 PRs pending, 53 forks, and currently broken build. I'd think hard before using it. – Fugleman

From the readme: (pytest-timeout) is not designed for precise timings or performance regressions. 🤔 – Dempstor

Answer 2

Bash functionality can be used:

EXIT_CODE=0
TIME_LIMIT=60

timeout $TIME_LIMIT pytest ... || EXIT_CODE=$?

if [ $EXIT_CODE -ne 0 ]; then
    
    echo "Your error message to log"
    
    ...
    
    exit $EXIT_CODE
    
fi

Bronk

Answer 3

Here is a lightweight way that you can use in a doctest or directly in your script and without any extra dependency, inspired from a question on how to limit execution time for a function

import signal
from contextlib import contextmanager


@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutError
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)

def efficiency_test(function, *args, **kwargs):
    try:
        with time_limit(2):
            return (True, function(*args, **kwargs))
    except TimeoutError as e:
        return (False, None)

With example usage:

import signal
from contextlib import contextmanager


@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutError
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)

def simple(a, b):
    return a + b

def complex_(a, b):
    for i in range(100000000):
        a += b
    return a

def efficiency_test(function, *args, **kwargs):
    try:
        with time_limit(2):
            return (True, function(*args, **kwargs))
    except TimeoutError as e:
        return (False, None)

if __name__ == "__main__":
    print(efficiency_test(simple, 1, 2))
    print(efficiency_test(complex_, 1, 2))

With output:

(True, 3)
(False, None)

Eschew