How to run different Pytest arguments or marks from VS Code test runner interface?

Question

I'm having trouble getting the VS Code PyTest code runner to work the way I'd like. It seems pytest options may be an all-or-nothing situation.

Is there any way to run different sets of PyTest options easily in the VS Code interface?

For example:

1. By default, run all tests not marked with @pytest.mark.slow.

   • This can be done with the argument -m "not slow"
   • But, if I put that in a pytest.ini file, then it will never run any tests marked slow, even if I pick that particular test in the interface and try to run it. The resulting output is collected 1 item... 1 item deselected.

2. Run sometimes with coverage enabled, and sometimes without.

The only way I can see to do this is to run PyTest from the command line, which then loses the benefit of auto-discovery, running/debugging individual tests from the in-line interface, etc.

What am I missing?

Note: Currently using VS Code 1.45.1, Python 3.7.6, and PyTest 5.3.5

Orogeny

I logged an enhancement request in the vscode-python repo on github for anyone interested. If you like the idea, please vote on it. – Orogeny

Answer 1

You're not missing anything. There currently isn't a way to provide per-execution arguments to get the integration you want with the Test Explorer.

Liszt

Ugh, so if I want to generally exclude a test from running, then I can't run it directly from the UI (the arguments to exclude it always exclude it). So I would always have to run the suite from the command line to be able to run any slow tests individually... That's a bummer. – Orogeny

Please feel free to open a feature request at github.com/microsoft/vscode-python. – Liszt

@BrettCannon, Are you aware of any changes to this end? – Shepard

There's been no change. – Liszt

Answer 2

You can't do this directly in the Test interface (still, as of Jan 2023), but you can do it with commandline arguments or separate scripts that you run at a terminal.

So it forces you to remember some parts of pytest's commandline interface, or write script(s) to run pytest exactly how you want for different use cases, but that's the best I have figured out so far without updating vs code or an extension. I don't know a way to update the Test Explorer interface with the results of those runs, however.

For example, you can put the following into your pytest.ini file, which will setup all the coverage flags, but run by default without coverage enabled, but just adding the --cov flag will run all the tests with coverage:

pytest.ini:

[pytest]
minversion = 6.0
python_files = test*.py
python_classes = Test
python_functions = test_* *_test
testpaths = ./
addopts = --cov-branch --cov-report html --cov-report xml:coverage.xml --cov-report term --ignore=<folder to ignore when discovering tests>

With the above, if one runs:

• pytest

at the terminal from the repo root, pytest will run all tests without coverage, but using:

• pytest --cov

it will include coverage as well, applying all the coverage flags in the addopts property in the ini file. This assumes you have installed and configured all the appropriate plugins and files for coverage.

Similarly, one could use the other pytest commandline args to run only certain test file marks, folders, etc, or create bash scripts that allow consolidating different types of tests or sub-suites into specific commands.

Orogeny

Answer 3

I found a way to run specific test from the interface, bypassing marks that skip tests; while keeping the marks/skips when you run everything.

In the conftest.py where you setup the "skip", you can check if it's a single test being invoked, and check if it's from from vscode using the args. Simply don't skip if it's a single test, and it's being called from VS Code.

def pytest_collection_modifyitems(config, items):
    # Always run test if it's a single test invoked from vscode
    if len(items) == 1 and "vscode_pytest" in config.invocation_params.args:
        return
    if config.getoption("--runslow"):
        return
    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
    for item in items:
        if "slow" in item.keywords:
            item.add_marker(skip_slow)

You can also setup a custom param in VS Code settings.json under python.testing.pytestArgs such as --runsinglewithoutchecks, to avoid putting vscode specific args in the codebase, which is a bit cleaner.

Dehydrogenase