Run tests concurrently

Question

I would like to run several tests concurrently using asyncio (/curio/trio) and pytest, but I couldn't find any information on that. Do I need to schedule them myself? And if I do, is there a way to have a nice output that separates (sub-)tests cases?

Here is a small toy example for which I'm trying this:

import pytest
import time
import asyncio

pytestmark = pytest.mark.asyncio
expected_duration = 1
accepted_error = 0.1

async def test_sleep():
    start = time.time()
    time.sleep(expected_duration)
    duration = time.time() - start
    assert abs(duration-expected_duration) < accepted_error

async def test_async_sleep():
    start = time.time()
    await asyncio.sleep(expected_duration)
    duration = time.time() - start
    assert abs(duration-expected_duration) < accepted_error

Gomuti

Not sure if it uses asyncio, but have you looked at pytest-xdist? pypi.org/project/pytest-xdist – Counsellor

I don't really care about asyncio (curio or trio would be just fine), but I do care that the solution is asynchronous (non-blocking IO). To give a little more context I have a lot of web requests to test (say 50 to 100) and I would just like to have a solution that manage to test all of them at once (so the total test time duration is close to the duration of the longest test). On the other hand I have almost no CPU workload. That's why I would prefer asynchronous solutions instead of multi-threaded ones. – Gomuti

Answer 1

Unfortunately, the way pytest works internally, you can't really run multiple tests at the same time under the same call to trio.run/asyncio.run/curio.run. (This is also good in some ways – it prevents state from leaking between tests, and with trio at least it lets you configure trio differently for different tests, e.g. setting one test to use the autojump clock while another test doesn't.)

Definitely the simplest option is to use pytest-xdist to run tests in separate threads. You can still use async internally inside each test – all these async libraries support running different loops in different threads.

If you really need to use async concurrency, then I think you'll have to write a single pytest test function, and then inside that function do your own scheduling and concurrency. If you do it this way, then from pytest's perspective there's only one test, so it won't be easy to get nice per-test output. I guess you could try using pytest-subtests?

Consolata

Answer 2

There's https://github.com/willemt/pytest-asyncio-cooperative now.

Today 2020-03-25, the limitations are quite steep — you have to ensure your tests don't share anything (well, technically, don't share mutable state) and you can't use mock.patch (technically don't mock anything another test might use).

You can follow the discussion at https://github.com/pytest-dev/pytest-asyncio/issues/69, I believe it is hard, but possible to come up with a way mark each fixture to allow or disallow concurrent use, and schedule tests to preserve these restrictions.

Dibru

How did I missed this message, that looks very promising for my use case (mostly independent HTTP requests). In this particular case I don't even think test isolation is that much important because in the worst case you would simply have false negative. But maybe I'm wrong on that last part. Thanks for the update. – Gomuti

As of version 0.17.1 of pytest-asyncio-cooperative the limitations around shared state have been relaxed further. For example, it's possible to use mock.patch as long as you use a "fixture lock" to prevent race conditions. – Atomy

Answer 3

Using pytest-subtests as suggested by Nathaniel seems to be a viable solution. Here is how it may be solved using trio, it run subtests on every function whose name starts with io_.

import pytest
import sys
import trio
import inspect
import re
import time


pytestmark = pytest.mark.trio
io_test_pattern = re.compile("io_.*")


async def tests(subtests):

    def find_io_tests(subtests, ignored_names):
        functions = inspect.getmembers(sys.modules[__name__], inspect.isfunction)
        for (f_name, function) in functions:
            if f_name in ignored_names:
                continue
            if re.search(io_test_pattern, f_name):
                yield (run, subtests, f_name, function)

    async def run(subtests, test_name, test_function):
        with subtests.test(msg=test_name):
            await test_function()

    self_name = inspect.currentframe().f_code.co_name
    async with trio.open_nursery() as nursery:
        for io_test in find_io_tests(subtests, {self_name}):
            nursery.start_soon(*io_test)


accepted_error = 0.1

async def io_test_1():
    await assert_sleep_duration_ok(1)

async def io_test_2():
    await assert_sleep_duration_ok(2)

async def io_test_3():
    await assert_sleep_duration_ok(3)

async def io_test_4():
    await assert_sleep_duration_ok(4)

async def assert_sleep_duration_ok(duration):
    start = time.time()
    await trio.sleep(duration)
    actual_duration = time.time() - start
    assert abs(actual_duration - duration) < accepted_error

Running python -m pytest -v outputs:

============================ test session starts =============================
platform darwin -- Python 3.7.0, pytest-4.6.2, py-1.8.0, pluggy-0.12.0
plugins: asyncio-0.10.0, trio-0.5.2, subtests-0.2.1
collected 1 item

tests/stripe_test.py::tests PASSED                                     [100%]
tests/stripe_test.py::tests PASSED                                     [100%]
tests/stripe_test.py::tests PASSED                                     [100%]
tests/stripe_test.py::tests PASSED                                     [100%]
tests/stripe_test.py::tests PASSED                                     [100%]

========================== 1 passed in 4.07 seconds ==========================

That's not perfect as the percentage is only relative to the number of tests and not to the number of subtests (ie. io_* marked functions here), but that seems like a good start.

Also note that time.time() is used so it makes sense for both trio and asyncio but in a real use case trio.current_time() should be used instead.

The same tests could be achieved using asyncio, you would basically have to replace three things:

• pytestmark = pytest.mark.trio → pytestmark = pytest.mark.asyncio
• yield (run, subtests, f_name, function) → yield run(subtests, f_name, function)
• And finally the nursey loop should be replaced with something like:

await asyncio.gather(*find_io_tests(subtests, {self_name}))

Gomuti