Python can't find my module

Question

I have a python project (which I run within a virtualenv) and that has the following structure:

Project
├───.git
├───venv
└───src
    ├───__init__.py
    ├───mymodules
    │   ├───__init__.py
    │   ├───module1.py
    │   └───module2.py
    └───scripts
        ├───__init__.py
        └───script.py

script.py

import src.mymodules.module1
...

---

I run the project with venv activated and from the Project directory using the following command:

(venv)$ python src/scripts/script.py

The script runs but gives out the following error before exiting:

Traceback (most recent call last):
  File "src/scripts/script.py", line 1, in <module>
    import src.mymodules.module1
ImportError: No module named src.mymodules.module1

I have tried running the python shell and trying to import the module from there and it gave no errors. I have _ _init__.py in every directory within src. Is python considering the working directory to be src/scripts? Why is that happening and how can I make src the working directory if that's the case?

Kempf

Python 2 or 3? 3 has a different way of handling package module imports. – Eduino

This might help #33773702 – Fadil

I've tried sys.path.append('/src') and sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))) still to no avail. – Kempf

Answer 1

Essentially, when you execute script.py directly, it doesn't know that it's part of a submodule of src, nor does it know where a module named src might be. This is the case in either python 2 or 3.

As you know, Python finds modules based on the contents of sys.path. In order to import any module, it must either be located in a directory that's listed in sys.path, or, in the same directory as the script that you're running.

When you say python src/scripts/script.py, sys.path includes the Project/src/scripts/ (because that's where script.py is located), but not Project. Because Project isn't in the path, the modules in that directory (src) aren't able to be imported.

To fix this:

I'm assuming that your script.py is an entry point for your src module (for example, maybe it's the main program). If that's true, then you could fix it by moving script.py up to the same level as src:

Project
├───.git
├───venv
|───script.py       <--- script.py moves up here
└───src
    ├───__init__.py
    └───mymodules
        ├───__init__.py
        ├───module1.py
        └───module2.py

This way, script.py can freely import anything in src, but nothing in src can import script.py.

If that's not the case, and script.py really is a part of src, you can use python's -m argument to execute script.py as part of the src module like so:

$ python -m src.scripts.script

Because you've told python which module you're running (src), it will be in the path. So, script.py will be aware that it's a submodule of src, and then will be able to import from src.

Be careful in this situation though - there's potential to create a circular import if something in src imports src.scripts.script.

---

As an alternative to both of these approaches, you can modify the sys.path directly in script.py:

import sys
sys.path.insert(0, '/path/to/Project') # location of src 

While this works, it's not usually my preference. It requires script.py to know exactly how your code is laid out, and may cause import confusion if another python program ever tries to import script.py.

Seko

Thanks, all 3 options work. The first one isn't an option for this project. The third one is not very elegant for the reason you stated. For now, I think I'll be using python -m src.scripts.script. I thought it would be enough to structure it like a package, as I've seen stated in other questions here? – Kempf

Python doesn't go "up" directory levels to find packages for various reasons. And submodules can and do import siblings, but mostly packages are intended to be used by other programs (and installed into a system package directory), not necessarily to be programs in themselves. This part of python can be confounding, but it does make some sense. – Seko

python -m src.scripts.script approach really helped me out. Thanks a lot! – Belgian

the most clear and straight answer I've see for the python import myth, instead of explicitly adding sys.path.insert... to source code, could use PYTHONPATH=/path/to/Project python script.py to get same goal. – Gogetter

@Shawn, good point. Setting the PYTHONPATH environment variable removes the script's need to know where it's located on disk (which will cause it to break if anything in the path is renamed). – Seko

Answer 2

Project
├───.git
├───venv
└───src
    ├───__init__.py
    ├───mymodules
    │   ├───__init__.py
    │   ├───module1.py
    │   └───module2.py
    └───scripts
        ├───__init__.py
        └───script.py

Alternatively you can import like the following in your script.py

import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__),'../../'))
import src.mymodules.module1

Now you can run script.py file from any location.

e.g :
python script.py
python /path to folder/script.py

Satyr

Your solution makes total sense, but doesn't seem to be working for me. Any thoughts? A RegressionTest folder contains an entrypoint file, say 'test.py' and another moduel 'check_results.py'. I want to import a utilities module from ../utils/CrossPlatform, so I do sys.path.append(os.path.dirname(os.path.dirname(file))) in the test.py, which then imports check_results.py. In check_results, sys.path yields the top level package that I'd expect. But, check_results cannot find utils (from utils.CrossPlatform import myutil.py). All folders have init,py. Thanks. – Orwin

Nuts. The issue was a conflicting module name somewhere in anaconda install path which is also called scripts and has init.py. – Orwin

Answer 3

Another solution is creating file some_file.pth and write your project dir (mother dir of your src). Put this file in /virtual_env/lib/pythonXX/site-packages/. In this way, you don't need to import sys.path in your script.

Arleenarlen

Thank you! this worked great. Do you happen to know what this does under the hood? – Barmen

Answer 4

If you face this problem when dealing with Pytest or coverage. Adding __init__.py file solve most of the cases.

Brilliance

Answer 5

Sometimes you saved the file without the ".py" extension

As we think of the name amodule, specially if you are from the Linux world, you may have saved the file as 'amodule' instead of 'amodule.py'

Martella