What sets up sys.path with Python, and when?

Question

When I run

import sys 
print sys.path

on my Mac (Mac OS X 10.6.5, Python 2.6.1), I get the following results.

/Library/Python/2.6/site-packages/ply-3.3-py2.6.egg
...
/Library/Python/2.6/site-packages/ipython-0.10.1-py2.6.egg
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages

/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Python/2.6/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode

They are grouped into 5 categories.

• /Library/Python/2.6/site-packages/*.egg
• /Library/Python/2.6/site-packages
• Frameworks/Python.framework/Versions/2.6/lib/python2.6
• Frameworks/Python.framework/Versions/2.6/Extras/lib/python
• PATH from PYTHONPATH environment variable.

And I can add more paths using the code

sys.path.insert(0, MORE_PATH)

• What routines sets up those paths, and when?
• Are some of the paths are built in python source code?
• Is it possible that the paths inserted with 'sys.path.insert' are ignored? I'm curious about this, as with mod_wsgi, I found the paths are not found with 'sys.path.insert'. I asked another post for this question.

ADDED

Based on Michael's answer, I looked into site.py, and I got the following code.

def addsitepackages(known_paths):
    """Add site-packages (and possibly site-python) to sys.path"""
    sitedirs = []
    seen = []

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.append(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitedirs.append(os.path.join(prefix, "Lib", "site-packages"))
        elif sys.platform == 'darwin' and prefix == sys.prefix:
            sitedirs.append(os.path.join("/Library/Python", sys.version[:3], "site-packages"))

I also think that the directory name that has site.py (/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6 for my Mac) should be built into Python source code.

Timisoara

Part of it can be answered with Python's sys.path value? – Overvalue

If your still trying to figure out your previous question - It is far more likely that your problem lies in the start up file, than something to do with site imports. It would be extremely poor practice for a package to remove paths they new nothing about. – Exactitude

Why seen is not a set ? :P – Stavropol

The best answer I found to the same question can be found here, which is a link to an answer to another SO question. – Hopi

Answer 1

Most of the stuff is set up in Python's site.py which is automatically imported when starting the interpreter (unless you start it with the -S option). Few paths are set up in the interpreter itself during initialization (you can find out which by starting python with -S).

Additionally, some frameworks (like Django I think) modify sys.path upon startup to meet their requirements.

The site module has a pretty good documentation, a commented source code and prints out some information if you run it via python -m site.

Armstrong

Apparently in Debian and derivatives, directories in /usr/local/lib/python<version>/dist-packages and /usr/{lib,share}/python<version>/dist-packages are automatically added to the path (according to /usr/lib/python2.7/site.py in Ubuntu 14.04). – Giesecke

Answer 2

From Learning Python:

sys.path is the module search path. Python configures it at program startup, automatically merging the home directory of the top-level file (or an empty string to designate the current working directory), any PYTHONPATH directories, the contents of any .pth file paths you've created, and the standard library directories. The result is a list of directory name strings that Python searches on each import of a new file.

Caravette

Answer 3

site.py is indeed the answers. I wanted to remove any dependencies on the old Python that is installed by default on my mac. This works pretty good, as 'site.py' is called each time the python interpreter is started.

For Mac, I manually added the following line at the end of main() in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/site.py:

    sys.path =  filter (lambda a: not a.startswith('/System'), sys.path)

Jaquesdalcroze

Please remember that site.py can be prevented from running by using command line -S option when running Python. From docs – Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. – Yuletide

Answer 4

Path has these parts:

• OS paths that have your system libraries
• current directory python started from
• environmental variable $PYTHONPATH
• you can add paths at runtime.

Paths are not ignored. But, they may not be found and that will not raise an error. sys.path should only be added too, not subtracted from. Django would not remove paths.

Exactitude

Answer 5

Adding to the accepted answer, and addressing the comments that say a module shouldn't remove entries from sys.path:

This is broadly true but there are circumstances where you might want to modify sys.path by removing entries. For instance - and this is Mac-specific; *nix/Windows corollaries may exist - if you create a customised Python.framework for inclusion in your own project you may want to ignore the default sys.path entries that point at the system Python.framework.

You have a couple of options:

1. Hack the site.py, as @damirv indicates, or

2. Add your own sitecustomize module (or package) to the custom framework that achieves the same end result. As indicated in the site.py comments (for 2.7.6, anyway):

   After these path manipulations, an attempt is made to import a module named sitecustomize, which can perform arbitrary additional site-specific customizations. If this import fails with an ImportError exception, it is silently ignored.

Grubstake

Answer 6

Also note: if the PYTHONHOME env var is set, standard libraries will be loaded from this path instead of the default, as documented.

This is not a direct answer to the question, but something I just discovered that was causing the wrong standard libraries to be loaded, and my searches lead me here along the way.

Tippett

PYTHONHOME doesn't really apply in case of mod_wsgi as Apache isn't going to be started from the user environment and the root user it starts as from system unit scripts wouldn't have it set. – Casmey

@GrahamDumpleton Maybe, but the title of the question is likely to lead others in more general cases here. I'm just hoping this information may help someone. – Tippett

Answer 7

You are using system python /usr/bin/python.

sys.path is set from system files at python startup.

Do not touch those files, in particular site.py, because this may perturb the system.

However, you can change sys.path within python, in particular, at startup :

in ~/.bashrc or ~/.zshrc:

export PYTHONSTARTUP=~/.pythonrc

in ~/.pythonrc:

write your changes to sys.path.

Those changes will be only for you in interactive shells.

For hacking at little risk for the system, install you own and more recent python version.

Baxter

how do you tell "You are using system python /usr/bin/python."? – Hosfmann

Give an example of "write your changes to sys.path." – Blockage