Expand Python Search Path to Other Source

Question

I have just joined a project with a rather large existing code base. We develop in linux and do not use and IDE. We run through the command line. I'm trying to figure out how to get python to search for the right path when I run project modules. For instance, when I run something like:

python someprojectfile.py

I get

ImportError: no module named core.'somemodule'

I get this for all of my imports to I assume it's an issue with the path.

How do I get Python to search ~/codez/project/ and all the files and folders for *.py files during import statements?

Ballot

Answer 1

There are a few possible ways to do this:

• Set the environment variable PYTHONPATH to a colon-separated list of directories to search for imported modules.
• In your program, use sys.path.append('/path/to/search') to add the names of directories you want Python to search for imported modules. sys.path is just the list of directories Python searches every time it gets asked to import a module, and you can alter it as needed (although I wouldn't recommend removing any of the standard directories!). Any directories you put in the environment variable PYTHONPATH will be inserted into sys.path when Python starts up.
• Use site.addsitedir to add a directory to sys.path. The difference between this and just plain appending is that when you use addsitedir, it also looks for .pth files within that directory and uses them to possibly add additional directories to sys.path based on the contents of the files. See the documentation for more detail.

Which one of these you want to use depends on your situation. Remember that when you distribute your project to other users, they typically install it in such a manner that the Python code files will be automatically detected by Python's importer (i.e. packages are usually installed in the site-packages directory), so if you mess with sys.path in your code, that may be unnecessary and might even have adverse effects when that code runs on another computer. For development, I would venture a guess that setting PYTHONPATH is usually the best way to go.

However, when you're using something that just runs on your own computer (or when you have nonstandard setups, e.g. sometimes in web app frameworks), it's not entirely uncommon to do something like

import sys
from os.path import dirname
sys.path.append(dirname(__file__))

Robichaux

So if I had like say 15 subdirectories, I would have to individually add each one? – Ballot

and could you give an example of a command line argument to change PYTHONPATH? – Ballot

To set PYTHONPATH: in .bashrc or whatever startup file your shell uses (if it's not Bash), write export PYTHONPATH=$PYTHONPATH:$HOME/codez/project. But if you have a bunch of subdirectories, I'd make a .pth file and use site.addsitedir. You can create a module sitecustomize that can call the function for you; try putting it at ~/.local/lib/python2.6/sitecustomize.py (substitute your Python version) so it hopefully gets automatically imported. – Robichaux

I put the following in my .bashrc file and I'm still having no luck with those imports. Any ideas? How would I make a .pth file anyway? export PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/codez/ export PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/codez/project export PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/codez/project/core export PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/codez/project/proxies export PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/codez/project/conf – Ballot

Try opening up a terminal and running echo $PYTHONPATH. If the environment variable has been set properly, you should see a colon-separated list of directories. For information about .pth files, see the documentation for the site module which I linked to in my answer. It tells you what the contents should be and how to use them. – Robichaux

But surely, there should be [..] only one. – Compatible

Answer 2

You should also read about python packages here: http://docs.python.org/tutorial/modules.html.

From your example, I would guess that you really have a package at ~/codez/project. The file __init__.py in a python directory maps a directory into a namespace. If your subdirectories all have an __init__.py file, then you only need to add the base directory to your PYTHONPATH. For example:

PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/project

In addition to testing your PYTHONPATH environment variable, as David explains, you can test it in python like this:

$ python
>>> import project                      # should work if PYTHONPATH set
>>> import sys
>>> for line in sys.path: print line    # print current python path

...

Olette

Answer 3

I know this thread is a bit old, but it took me some time to get to the heart of this, so I wanted to share.

In my project, I had the main script in a parent directory, and, to differentiate the modules, I put all the supporting modules in a sub-folder called "modules". In my main script, I import these modules like this (for a module called report.py):

from modules.report import report, reportError

If I call my main script, this works. HOWEVER, I wanted to test each module by including a main() in each, and calling each directly, as:

python modules/report.py

Now Python complains that it can't find "a module called modules". The key here is that, by default, Python includes the folder of the script in its search path, BUT NOT THE CWD. So what this error says, really, is "I can't find a modules subfolder". The is because there is no "modules" subdirectory from the directory where the report.py module resides.

I find that the neatest solution to this is to append the CWD in Python search path by including this at the top:

import sys

sys.path.append(".")

Now Python searches the CWD (current directory), finds the "modules" sub-folder, and all is well.

Osbourne

Answer 4

The easiest way I find is to create a file any_name.pth and put it in your folder \Lib\site-packages. You should find that folder wherever python is installed.

In that file, put a list of directories where you want to keep modules for importing. For instance, make a line in that file like this:

C:\Users\example\...\example

You will be able to tell it works by running this in python:

import sys
for line in sys.path:
    print line

You will see your directory printed out, amongst others from where you can also import. Now you can import a mymodule.py file that sits in that directory as easily as:

import mymodule

This will not import subfolders. For that you could imagine creating a python script to create a .pth file containing all sub folders of a folder you define. Have it run at startup perhaps.

Lidia

Answer 5

I read this question looking for an answer, and didn't like any of them.

So I wrote a quick and dirty solution. Just put this somewhere on your sys.path, and it'll add any directory under folder (from the current working directory), or under abspath:

#using.py

import sys, os.path

def all_from(folder='', abspath=None):
    """add all dirs under `folder` to sys.path if any .py files are found.
    Use an abspath if you'd rather do it that way.

    Uses the current working directory as the location of using.py. 
    Keep in mind that os.walk goes *all the way* down the directory tree.
    With that, try not to use this on something too close to '/'

    """
    add = set(sys.path)
    if abspath is None:
        cwd = os.path.abspath(os.path.curdir)
        abspath = os.path.join(cwd, folder)
    for root, dirs, files in os.walk(abspath):
        for f in files:
            if f[-3:] in '.py':
                add.add(root)
                break
    for i in add: sys.path.append(i)

>>> import using, sys, pprint
>>> using.all_from('py') #if in ~, /home/user/py/
>>> pprint.pprint(sys.path)
[
#that was easy
]

And I like it because I can have a folder for some random tools and not have them be a part of packages or anything, and still get access to some (or all) of them in a couple lines of code.

Professor

Answer 6

the first task is to find out where the current environment gets its standard libraries from. And that's where I would store my OOP modules.

Import numpy

numpy.__file__

You get a catalog on your environment. For another computer, it is a good idea to create a separate directory for OOP modules.

Import sys 

sys.path.append('F:/Python/My_libra/')

Waltz

Answer 7

New option for old question.
Installing fail2ban package on Debian, looks like it's hardcoded to install on /usr/lib/python3/dist-packages/fail2ban a path not on python3 sys.path.

> python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Jun 25 2019, 18:51:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
>>>

so, instead of just copying, I (bash) linked the library to newer versions.
Future updates to the original app, will also be automatically applied to the linked versions.

 if [ -d /usr/lib/python3/dist-packages/fail2ban ]
   then
      for d in /usr/lib/python3.*
      do
         [ -d ${d}/fail2ban ] || \
            ln -vs /usr/lib/python3/dist-packages/fail2ban ${d}/
      done
   fi

Fusible