python: Interplay between lib/site-packages/site.py and lib/site.py

Question

Due to a specific problem which I managed to solve, I spent most of today figuring out how site.py(s) work. There is a point which I don't understand.

As far as I understand, when python is loaded, first lib/python2.7/site-packages/site.py is run. It goes over PYTHONPATH, searches for lib/python2.7/site.py, and imports it. This file has the addsitedir method, that not only adds a path to sys.path, but also processes *.pth files which exists on it. At this point, main() from lib/python2.7/site.py is ran, and addsitedir is ran on site-packages and on user-site-packages.

Now comes the weird part. Now we go back to lib/python2.7/site-packages/site.py, which goes over each path in pythonpath, and runs addsitedir on it. I find this weird for two reasons:

1. addsitedir is ran on lib/python2.7/site-packages twice.
2. This is not so bad in itself (nothing can be added to sys.path twice), but it seems that lib/python2.7/site.py has a mechanism for allowing the ambitious user to manipulate sys.path by implementing a usercustomize module (hey, it's even in the docs). Obviously, when you implement such a mechanism, you want to make sure the user gets in last, so he can have control over everything added to sys.path. But this is not the case here (as I was frustrated to find out). Most likely, the second call to lib/python2.7/site-packages will override everything done in usercustomize.

I know it's terrible, but I added a print statement to addsitedir, printing the path it receives, so I could show what's going on. These are the paths processed:

/home/user/.local/lib/python2.7/site-packages #lib/python2.7/site.py
/home/user/py/lib/python2.7/site-packages     #lib/python2.7/site.py
#This is where your usercustomize runs
#Followin calls are from lib/python2.7/site-packages/site.py
/home/user/py/lib/python2.7/site-packages/numpy-1.9.0-py2.7-linux-x86_64.egg
/home/user/Develop/Python/myproject
/home/user/lmfit-0.7.2
/home/user/py/lib/python2.7/site-packages #NOTE: this runs a second time

So what am I asking here? :)

A. I'd appreciate insights as to why the second call to site-packages is needed.

B. Is usercustomize indeed limited as I believe it is due to this implementation? Considering this, how would you implement something removing paths from sys.path (theoretically)?

---

Requested debug output:

:genie39:~ ;-) python2.7 -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /home/user/py/lib/python2.7/site-packages/site.pyc matches /home/user/py/lib/python2.7/site-packages/site.py
import site # precompiled from /home/user/py/lib/python2.7/site-packages/site.pyc
# /home/user/py/lib/python2.7/os.pyc matches /home/user/py/lib/python2.7/os.py
import os # precompiled from /home/user/py/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /home/user/py/lib/python2.7/posixpath.pyc matches /home/user/py/lib/python2.7/posixpath.py
import posixpath # precompiled from /home/user/py/lib/python2.7/posixpath.pyc
# /home/user/py/lib/python2.7/stat.pyc matches /home/user/py/lib/python2.7/stat.py
import stat # precompiled from /home/user/py/lib/python2.7/stat.pyc
# /home/user/py/lib/python2.7/genericpath.pyc matches /home/user/py/lib/python2.7/genericpath.py
import genericpath # precompiled from /home/user/py/lib/python2.7/genericpath.pyc
# /home/user/py/lib/python2.7/warnings.pyc matches /home/user/py/lib/python2.7/warnings.py
import warnings # precompiled from /home/user/py/lib/python2.7/warnings.pyc
# /home/user/py/lib/python2.7/linecache.pyc matches /home/user/py/lib/python2.7/linecache.py
import linecache # precompiled from /home/user/py/lib/python2.7/linecache.pyc
# /home/user/py/lib/python2.7/types.pyc matches /home/user/py/lib/python2.7/types.py
import types # precompiled from /home/user/py/lib/python2.7/types.pyc
# /home/user/py/lib/python2.7/UserDict.pyc matches /home/user/py/lib/python2.7/UserDict.py
import UserDict # precompiled from /home/user/py/lib/python2.7/UserDict.pyc
# /home/user/py/lib/python2.7/_abcoll.pyc matches /home/user/py/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /home/user/py/lib/python2.7/_abcoll.pyc
# /home/user/py/lib/python2.7/abc.pyc matches /home/user/py/lib/python2.7/abc.py
import abc # precompiled from /home/user/py/lib/python2.7/abc.pyc
# /home/user/py/lib/python2.7/_weakrefset.pyc matches /home/user/py/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /home/user/py/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /home/user/py/lib/python2.7/copy_reg.pyc matches /home/user/py/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /home/user/py/lib/python2.7/copy_reg.pyc
import imp # builtin
# /home/user/py/lib/python2.7/site.pyc matches /home/user/py/lib/python2.7/site.py
import site # precompiled from /home/user/py/lib/python2.7/site.pyc
# /home/user/py/lib/python2.7/traceback.pyc matches /home/user/py/lib/python2.7/traceback.py
import traceback # precompiled from /home/user/py/lib/python2.7/traceback.pyc
# /home/user/py/lib/python2.7/sysconfig.pyc matches /home/user/py/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /home/user/py/lib/python2.7/sysconfig.pyc
# /home/user/py/lib/python2.7/re.pyc matches /home/user/py/lib/python2.7/re.py
import re # precompiled from /home/user/py/lib/python2.7/re.pyc
# /home/user/py/lib/python2.7/sre_compile.pyc matches /home/user/py/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /home/user/py/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /home/user/py/lib/python2.7/sre_parse.pyc matches /home/user/py/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /home/user/py/lib/python2.7/sre_parse.pyc
# /home/user/py/lib/python2.7/sre_constants.pyc matches /home/user/py/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /home/user/py/lib/python2.7/sre_constants.pyc
# /home/user/py/lib/python2.7/_sysconfigdata.pyc matches /home/user/py/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /home/user/py/lib/python2.7/_sysconfigdata.pyc
# zipimport: found 604 names in /home/user/py/lib/python2.7/site-packages/pytz-2014.7-py2.7.egg
# zipimport: found 20 names in /home/user/py/lib/python2.7/site-packages/hashlib-20081119-py2.7-linux-x86_64.egg
# zipimport: found 40 names in /home/user/py/lib/python2.7/site-packages/pysqlite-2.6.3-py2.7-linux-x86_64.egg
# zipimport: found 7 names in /home/user/py/lib/python2.7/site-packages/mock-1.0.1-py2.7.egg
import encodings # directory /home/user/py/lib/python2.7/encodings
# /home/user/py/lib/python2.7/encodings/__init__.pyc matches /home/user/py/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /home/user/py/lib/python2.7/encodings/__init__.pyc
# /home/user/py/lib/python2.7/codecs.pyc matches /home/user/py/lib/python2.7/codecs.py
import codecs # precompiled from /home/user/py/lib/python2.7/codecs.pyc
import _codecs # builtin
# /home/user/py/lib/python2.7/encodings/aliases.pyc matches /home/user/py/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /home/user/py/lib/python2.7/encodings/aliases.pyc
# /home/user/py/lib/python2.7/encodings/utf_8.pyc matches /home/user/py/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /home/user/py/lib/python2.7/encodings/utf_8.pyc
Python 2.7.8 (default, Sep  7 2014, 12:14:33) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/home/user/py/lib/python2.7/site-packages/readline-6.2.4.1-py2.7-linux-x86_64.egg/readline.so", 2);
import readline # dynamically loaded from /home/user/py/lib/python2.7/site-packages/readline-6.2.4.1-py2.7-linux-x86_64.egg/readline.so
>>> 

Output of python -vv is here

Kauffman

Answer 1

The lib/python2.7/site-packages/site.py file is not normally loaded. That's because it is lib/python2.7/site.py's job to add the site-packages paths to sys.path to begin with, and the site.py in site-packages is simply not visible. If you have a site.py in site-packages then that is an error, there should be no such file there.

What happens in an unpatched Python without is:

• Python starts, with a limited sys.path. site-packages is not part of this list, unless you set a PYTHONPATH variable that includes it anyway.
• Python imports site.py, it'll import the one listed first on sys.path.
• lib/python2.7/site.py is found and loaded
• site.py adds site-packages to sys.path

That's it, no further site.py modules are loaded. Even if you tried, it'd find the module that was already imported; sys.modules['site'] exists and holds objects loaded from lib/python2.7/site.py.

Your installation, however, has an older setuptools installed, and it includes a special version of site.py, which the easy_install command will install into site-packages if not yet present. It'll load the original site.py by explicitly scanning the original sys.path with any PYTHONPATH-supplied paths ignored and loading the original site.py module manually using the imp.find_module() and imp.load_module() low-level functions, thus bypassing the normal module cache.

Its intent was to alter the sys.path order to give PYTHONPATH-listed .pth files a higher precedence, see the original commit adding the patch:

Note: this version includes a hacked 'site.py' to support processing .pth files in directories that come before site-packages on sys.path.

The patch has been removed entirely from more recent setuptools releases, as early as 2006 in the original setuptools.

So, either your Linux distribution has been set up to add lib/python2.7/site-packages to your PYTHONPATH or your shell has this set up for you, or your Python has been patched to include it, and you have the old setuptools 'patch' in your site-packages.

It is entirely safe to remove that file.

Maryleemarylin

This answer is definitive and extremely helpful. Most likely, site-packages/site.py came from installing setuptools. The old way of installing it to a custom Python installation required setting PYTHONPATH. However, easy_install no longer requires that. "[A]ll of these [older ways] are effectively deprecated by the User scheme brought in by PEP-370 in Python 2.6." – Impeller