Test discovery failure when tests in different directories are called the same

Question

Using py.test, two tests called the same in different directory causes py.test to fail. Why is that? How can I change this without renaming all the tests?

To duplicate do:

; cd /var/tmp/my_test_module
; mkdir -p ook/test           
; mkdir -p eek/test
; touch ook/test/test_proxy.py
; touch eek/test/test_proxy.py
; py.test
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.4
collected 0 items / 1 errors 

==================================== ERRORS ====================================
___________________ ERROR collecting ook/test/test_proxy.py ____________________
import file mismatch:
imported module 'test_proxy' has this __file__ attribute:
  /home/ygolanski/code/junk/python/mymodule/eek/test/test_proxy.py
which is not the same as the test file we want to collect:
  /home/ygolanski/code/junk/python/mymodule/ook/test/test_proxy.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
=========================== 1 error in 0.01 seconds ============================

Gelation

Answer 1

Putting an __init__.py is one way of resolving the conflict. Unlike nose, current pytest does not try to unload test modules in order to import test modules with the same import name. I used to think it's a bit magic to do this auto-unimporting and might mess up people's expectation from what the import mechanism does; sometimes people rely on the global state of a test module and with auto-unloading you lose it (a test module importing from another test module might then do unexpected things). But maybe it's not a practical issue and thus pytest could add a similar hack ...

Lineation

I agree that requiring an init.py makes sense. If a test is not in a package, then it is essentially a top-level module (in the OP, test_proxy) and there should be only one. By putting the test modules in relevant packages (ook and eek), it provides a proper namespacing of the tests. I say status quo is best. It might alleviate some pain to have the error message link to this question or to something in the docs that explains the reasoning and the technique to work around the issue. – Berber

Just a note that py.test docs specifically advise against putting __init__.py in test directories: "avoid __init__.py files in your test directories. This way your tests can run easily against an installed version of mypkg, independently from if the installed package contains the tests or not". Taken from pytest.org - Good Integration Practices. – Lastex

Update: The recommendation in @famousgarkin's comment above and answer (https://mcmap.net/q/257822/-test-discovery-failure-when-tests-in-different-directories-are-called-the-same) appears to no longer be in the docs (at least searching for "avoid" doesn't bring up the quote above): docs.pytest.org/en/latest/…. In fact, the examples in that link show __init__.py in test directories, so it seems that the accepted answer is the correct one. – Volcano

I had the same problem, but in my case was because had added from __future__ import absolute_import on some of my test files, removing that fixed the issue. – Hobbes

I have never full understood pytest test discovery logic. When I experience a discovery error I just look for places to put __ini__.pys ¯_(ツ)_/¯. Has worked for me thus far. – Tanto

@Lastex removed in github.com/pytest-dev/pytest/pull/2297 – Liz

Answer 2

This is an actual feature of py.test. You can find the reason for this behavior stated in pytest.org - Good Integration Practices - Choosing a test layout / import rules:

• avoid __init__.py files in your test directories. This way your tests can run easily against an installed version of mypkg, independently from if the installed package contains the tests or not.

As that's the recommended workflow of working with py.test: install the package under development with pip install -e, then test it.

Because of this, I myself opt for unique test names, in the convention over configuration manner. It also ensures that you don't get ambiguous test names in the various test run output.

If you need to keep the test names and don't care about the above mentioned functionality, you should be ok with putting an __init__.py.

Lastex

I don't get the use case: "This way your tests can run easily against an installed version of mypkg". I run my tests against my development version of mypkg. This way the tests exist. I create __init__.py files to avoid this "unique basename" error message. – Tesch

I created a feature request to change the docs: bitbucket.org/hpk42/pytest/issue/529/… – Tesch

@Tesch The use case is when you want to test your package as installed via setup.py. This can be used to ensure you're including all required files in the install, like bundled data and that all dependencies are handled correctly. It can also be used to install on environments where the target system does not have a compiler and you're going to use a binary egg or wheel for the installation. – Reductive

there's a typo in the link. here's the correct link: pytest.org/latest/… – Vizcacha

Answer 3

I had the same error, but the solution had nothing to do with the init files nor the name on the test files. I had different python versions on my macbook and on my Docker container. I launched the tests once in the bash from the macbook at the root of the project, instead of the bash of the container.

The solution was to remove the wrongly created files by running (from the bash of the container):

find -name '*.pyc' -delete
find -name __pycache__ -delete

Then launch the test again (still from the bash of the container) and everything worked just fine:

py.test

Feinberg

I had the same problem inside a docker container and this answer saved my life. I think it should be added to the accepted answer by @Lineation . – Godbey

Answer 4

In this case, you can either create case/__init__.py and pytest/__init__.py files or rename the test files to unique names.

Ogrady