Importing correctly with pytest

Question

I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.

My directory structure is as follows:

src/
    main.py
    util.py
    test/
        test_util.py
    geom/
        vector.py
        region.py
        test/
            test_vector.py
            test_region.py

To run, I call python main.py from src/.

In main.py, I import both vector and region with

from geom.region import Region
from geom.vector import Vector

In vector.py, I import region with

from geom.region import Region

These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.

---

Some Problems and My Solution Attempts

My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":

import imp
util = imp.load_source("util", "util.py")

This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".

However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:

from geom.region import *
from region import *

These both give errors of the form

ImportError: No module named [geom.region / region]

I don't know what to do next to solve this problem; my understanding of imports in Python is limited.

What is the proper way to handle imports when using pytest?

---

Edit: Extremely Hacky Solution

In vector.py, I changed the import statement from

from geom.region import Region

to simply

from region import Region

This makes the import relative to the directory of "vector.py".

Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:

import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))

This enables Python to find "../region.py" from "geom/test/test_vector.py".

This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either

1) An import strategy that is compatible with pytest, or

2) An option in pytest that makes it compatible with my import strategy

So I am leaving this question open for answers of these kinds.

Fokos

I am still searching for that py.test compatible import strategy : / – Monied

@Zelphir python -m pytest tests/ works fine. – Wore

@Wore If I do that, in my project python doesn't find all the other imports used by the test cases (or the imports classes / modules use in the test cases). If I change those to be absolute imports, I can run the tests using your command, but when I try to run the program it doesn't find the imports – endless cycle of import statement changing and I don't know how to fix it. – Monied

Absolute imports are recommended in your main package too. See answers to https://mcmap.net/q/181888/-absolute-vs-explicit-relative-import-of-python-module/3565696. But remember to add __init__.py files in your directories, to create a regular package. (I never tried with a namespace package) – Wore

You mention, "These all work fine when I run the code in a standard run." Can you describe exactly how you start a "standard run"? That will let us figure out how sys.path is configured when a "standard run" starts. – Malignant

Answer 1

import looks in the following directories to find a module:

1. The home directory of the program. This is the directory of your root script. When you are running pytest your home directory is where it is installed (/usr/local/bin probably). No matter that you are running it from your src directory because the location of your pytest determines your home directory. That is the reason why it doesn't find the modules.
2. PYTHONPATH. This is an environment variable. You can set it from the command line of your operating system. In Linux/Unix systems you can do this by executing: 'export PYTHONPATH=/your/custom/path' If you wanted Python to find your modules from the test directory you should include the src path in this variable.
3. The standard libraries directory. This is the directory where all your libraries are installed.
4. There is a less common option using a pth file.

sys.path is the result of combining the home directory, PYTHONPATH and the standard libraries directory. What you are doing, modifying sys.path is correct. It is something I do regularly. You could try using PYTHONPATH if you don't like messing with sys.path

Julesjuley

Thanks! This cleared things up. Made a slight change to the solution I used though (edit shown above) – Fokos

Answer 2

The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)

Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.

There are some implications of this of which you should beware:

1. The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.

2. You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.

The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.

The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.

The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.

Malignant

The problem in my case was the missing __init__.py in the test folder. Thanks. – Stockist

It seems that every test folder(any level) shall have a __init__.py. – Whodunit

I also ran into this __init__.py everywhere issue :D LOL Putting an __init__.py in the tests/ folder saved me! – Hussein

thank you for this great explanation, I finally understand where to put __init__.py files and why!! – Reims

In my case after adding init.py under unit test folder it started working, can some one explain what is the reason for this behavior? – Murrah

I don't understand this, and it seems that there is some confusion because most of the comments are about __init__.py under the "tests" dir, whereas most of your talk seems to be about __init__.py in the applications (i.e. under the "src" dir). Adding __init__.py under every directory under and including "src" does not solve the extremely irksome problem of an application file not being able to import a sibling file. Any chance you might try explaining a little more about the steps to take to achieve this? – Zaid

@mikerodent My files containing tests are in the same directories as the source under test. There's almost never a good reason to use even a separate subdirectory, much less a separate directory hierarchy, unless you're trying to hide your tests from the people who should be updating them when they touch the code, and a parallel directory hierarchy with the same names produces the "which of the two directories is the package?" problem you're probably having. – Malignant

Unfortunately, you really need to understand the Python package system, particularly the difference between "regular" packages defined with __init__.py and "namespace" packages that do not have an __init__.py. (I have some notes here that may help.) I suggest your best bet is to avoid using namespace packages and use explicit (rather than relative) package names in all import statements. – Malignant

And you almost certainly do not want an __init__.py under src/; that creates a package called src implying that src/foo/bar.py is named not foo.bar but src.foo.bar. – Malignant

@Malignant Thanks for taking the trouble to answer. I used to be "on the other side" (Java, Gradle, Groovy), and Gradle by default has this separate directory structure. Functional/integration tests also don't necessarily correspond to one particular class or package of course. With a separate dirs structure, deployment to a production location is arguably simpler: just exclude the "tests" directory. But you've given me a valuable clue about why this is causing me and others so much grief :). – Zaid

@mikerodent Yeah, Java projects originally used separate trees because it made it easier to exclude test code from your JAR file when using less-capable build systems such as Ant. That usually doesn't apply to Python; since you're releasing source anyway, including the tests is usually not harmful and often a good thing. Both suffer from the issue that the fully qualified name is not set in the source file but is partly (for Java) or fully (for Python) determined by filesystem paths; Python introduces the extra complexity that these are easily varied at runtime. – Malignant

I've been struggling with this issue for a few months now and this is a fantastic explanation that helped me resolve my problems within a matter of minutes. Thanks!! – Tartarus

Answer 3

If you include an __init__.py file inside your tests directory, then when the program is looking to set a home directory it will walk 'upwards' until it finds one that does not contain an init file. In this case src/.

From here you can import by saying :

from geom.region import *

you must also make sure that you have an init file in any other subdirectories, such as the other nested test directory

Elwaine

Answer 4

I was wondering what to do about this problem too. After reading this post, and playing around a bit, I figured out an elegant solution. I created a file called "test_setup.py" and put the following code in it:

import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))

I put this file in the top-level directory (such as src). When pytest is run from the top-level directory, it will run all test files including this one since the file is prefixed with "test". There are no tests in the file, but it is still run since it begins with "test".

The code will append the current directory name of the test_setup.py file to the system path within the test environment. This will be done only once, so there are not a bunch of things added to the path.

Then, from within any test function, you can import modules relative to that top-level folder (such as import geom.region) and it knows where to find it since the src directory was added to the path.

If you want to run a single test file (such as test_util.py) instead of all the files, you would use:

pytest test_setup.py test\test_util.py

This runs both the test_setup and test_util code so that the test_setup code can still be used.

Wrath

Answer 5

Are so late to answer that question but usining python 3.9 or 3.10 u just need to add __init__.py folder in tests folders.

When u add this file python interprets this folders as a module. Wold be like this

src/
main.py
util.py
test/
    __init__.py
    test_util.py
geom/
    vector.py
    region.py
    test/
        __init__.py
        test_vector.py
        test_region.py

so u just run pytest.

Sorry my poor english

Devlen

Answer 6

Not the best solution, but maybe the fastest one:

cd path/python_folder

python -m pytest python_file.py

Prosser