PYTHONPATH vs. sys.path

Question

Another developer and I disagree about whether PYTHONPATH or sys.path should be used to allow Python to find a Python package in a user (e.g., development) directory.

We have a Python project with a typical directory structure:

Project
    setup.py
    package
        __init__.py
        lib.py
        script.py

In script.py, we need to do import package.lib. When the package is installed in site-packages, script.py can find package.lib.

When working from a user directory, however, something else needs to be done. My solution is to set my PYTHONPATH to include "~/Project". Another developer wants to put this line of code in the beginning of script.py:

sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

So that Python can find the local copy of package.lib.

I think this is a bad idea, as this line is only useful for developers or people running from a local copy, but I can't give a good reason why it is a bad idea.

Should we use PYTOHNPATH, sys.path, or is either fine?

Stature

Seems the votes and answers are split fairly evenly with a very slight lean towards using PYTHON_PATH though this could be sampling noise or unintentional biasing from the question. – Recor

For the difference between PATH and sys.path (and indirectly PYTHONPATH) see also https://mcmap.net/q/138350/-sys-path-vs-path – Worsham

Answer 1

If the only reason to modify the path is for developers working from their working tree, then you should use an installation tool to set up your environment for you. virtualenv is very popular, and if you are using setuptools, you can simply run setup.py develop to semi-install the working tree in your current Python installation.

Peculate

Can you offer a bit more clarification on this? Even if you stand up a conda / virtualenv environment, how would this put the toplevel dir on your python path? – Nathannathanael

Thanks for the helpful answer! Also, is this still current? Is Setuptools the recommended way to work on things like this? – Schatz

Answer 2

I hate PYTHONPATH. I find it brittle and annoying to set on a per-user basis (especially for daemon users) and keep track of as project folders move around. I would much rather set sys.path in the invoke scripts for standalone projects.

However sys.path.append isn't the way to do it. You can easily get duplicates, and it doesn't sort out .pth files. Better (and more readable): site.addsitedir.

And script.py wouldn't normally be the more appropriate place to do it, as it's inside the package you want to make available on the path. Library modules should certainly not be touching sys.path themselves. Instead, you'd normally have a hashbanged-script outside the package that you use to instantiate and run the app, and it's in this trivial wrapper script you'd put deployment details like sys.path-frobbing.

Bleb

The problem with site.addsitedir is that it does a append on sys.path, meaning that an installed package will take precedence over the local package in development (and hair pulling may ensue). sys.path.insert(0... is needed to overcome that. – Carpathoukraine

@EliBendersky: should be sys.path.insert(1. https://mcmap.net/q/137242/-why-use-sys-path-append-path-instead-of-sys-path-insert-1-path/125507 – Jospeh

isn't a better way to install the self-written module with pip than both PYTHONPATH change and os.sys ? – Counterpoison

Answer 3

In general I would consider setting up of an environment variable (like PYTHONPATH) to be a bad practice. While this might be fine for a one off debugging but using this as
a regular practice might not be a good idea.

Usage of environment variable leads to situations like "it works for me" when some one
else reports problems in the code base. Also one might carry the same practice with the test environment as well, leading to situations like the tests running fine for a particular developer but probably failing when some one launches the tests.

Ambition

isn't a better way to install the self-written module with pip than both PYTHONPATH change and os.sys ? – Counterpoison

If self written module is only for the usage in your project, one won't take steps make it pip installable. Not every module one writes need to be/will be available via pip. – Ambition

Answer 4

Along with the many other reasons mentioned already, you could also point outh that hard-coding

sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

is brittle because it presumes the location of script.py -- it will only work if script.py is located in Project/package. It will break if a user decides to move/copy/symlink script.py (almost) anywhere else.

Bandit

what if th euser decides to move the package instead of the script? would that harm the PYTHONPATH method? is there anything better than these 2 methods? – Counterpoison

Answer 5

Neither hacking PYTHONPATH nor sys.path is a good idea due to the before mentioned reasons. And for linking the current project into the site-packages folder there is actually a better way than python setup.py develop, as explained here:

pip install --editable path/to/project

If you don't already have a setup.py in your project's root folder, this one is good enough to start with:

from setuptools import setup
setup('project')

Novikoff

in order to install the package, should there be a setup.py in the package directory? and what is inside that setup.py? – Counterpoison

Answer 6

I think, that in this case using PYTHONPATH is a better thing, mostly because it doesn't introduce (questionable) unneccessary code.

After all, if you think of it, your user doesn't need that sys.path thing, because your package will get installed into site-packages, because you will be using a packaging system.

If the user chooses to run from a "local copy", as you call it, then I've observed, that the usual practice is to state, that the package needs to be added to PYTHONPATH manually, if used outside the site-packages.

Monikamoniker

Following a standard project directory structure I find that I'm going to have to do something to make imports in tests work for the normal pytest invocation from the project root. The least intrusive of all the hacks to make the "standard" test directory location work is updating PYTHONPATH in the venv activate script. I presume this is equivalent to doing what IDEs always do anyway. It seems IDEs can do whatever crap they like and no-one calls it a hack. If you use tox for your tests you don't need this but then your tests are at arms length. – Allista