How does Python keep track of modules installed with eggs?

Question

If I have a module, foo, in Lib/site-packages, I can just import foo and it will work. However, when I install stuff from eggs, I get something like blah-4.0.1-py2.7-win32.egg as a folder, with the module contents inside, yet I still only need do import foo, not anything more complicated. How does Python keep track of eggs? It is not just dirname matching as if I drop that folder into a Python installation without going through dist-utils, it does not find the module.

To be clearer: I just installed zope. The folder name is "zope.interface-3.3.0-py2.7-win32.egg". This works:

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zope.interface
>>>

I create a "blah-4.0.1-py2.7-win32.egg" folder with an empty module "haha" in it (and __init__.py). This does not work:

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import blah.haha
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named blah.haha
>>>

This does, though:

Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkg_resources import require
>>> require("blah>=1.0")
[blah 4.0.1 (c:\python27\lib\site-packages\blah-4.0.1-py2.7-win32.egg)]
>>> import haha
>>>

So how do I make it work without a require?

Pollux

The Python source is completely available to you. You can read up on the import statement's implementation without too much trouble. As they say: "Use the Source, Luke." – Ario

@S.Lott: any question can be answered by a different level of "do it yourself." that's not what this site is for. this is also more efficient. instead of me poring through the source and wasting an hour or so, making me unhappy, I can ask this here, check back once in a while, get my answer, be happy, give rep to someone else to give them their dopamine release, have this question appear on google if anyone else wants to know, etc. – Pollux

"wasting an hour"? Really? Wasting seems harsh. You would learn a lot. And, further, this part of the doco seems pretty complete. docs.python.org/library/modules.html#importing-modules. I'm surprised you didn't start with the documentation. – Ario

S.Lott: eggs and easy_install are not part of the Python standard library so just reading about importing modules isn't really going to answer the OP's question. – Cudweed

@S.Lott: true, wasting is harsh. i would learn something, as well. what i really meant to get at was comparative advantage. someone else who knows this already can spend less time answering my question than it would take me. plus, i still learn doing it this way, though not as well, i agree, since i didn't have to work so hard at it. i also agree that the second a question comes up one shouldn't immediately jump on SO but try it a bit themselves, first. but having said all that i feel asking this question was worth it. – Pollux

Answer 1

If you use the easy_install script provided by setuptools (or the Distribute fork of it) to install packages as eggs, you will see that, by default, it creates a file named easy-install.pth in the site-packages directory of your Python installation. Path configuration files are a standard feature of Python:

A path configuration file is a file whose name has the form package.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path.

easy_install makes heavy use of this Python feature. When you use easy_install to add or update a distribution, it modifies easy-install.pth to add the egg directory or zip file. In this way, easy_install maintains control of the module searching order and ensures that the eggs it installs appear early in the search order. Here is an example of the contents of an easy-install.pth:

import sys; sys.__plen = len(sys.path)
./appscript-0.21.1-py2.6-macosx-10.5-ppc.egg
./yolk-0.4.1-py2.6.egg
./Elixir-0.7.1-py2.6.egg
./Fabric-0.9.0-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginse
rt',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

As you can see here and if you examine the code in setuptools, you will find it goes to some trickery to bootstrap itself and then cover its tracks which can make debugging problems with site.py and interpreter startup a bit interesting. (That is one of the reasons that some developers are not fond of using it.)

If you use the -m parameter of easy_install to install a distribution as multi-version, the easy-install.pth entry for it is not added or is removed if it already exists. This is why the easy_install documentation tells you to use -m before deleting an installed egg.

Cudweed

I have never used the easy_install script, yet such an installation (an .egg file instead of all the files in the package) was automaticall created when I installed EasyAI uisng the standard way: pytnon -m pip install easyai. I have installed dozens of packages in this way and only this one created an .egg. But it doesn't bother me at all. In fact, I would like to know how to install more packages as .egg and even better, if I can convert some installed packages to '.egg'. – Australorp

Answer 2

When you run easy_install it copies the egg into site-packages and puts the path to that egg on your sys.path variable. (Note that sys.path is not your PATH environment variable, it is constructed from PYTHONPATH and other environment variables. So the .egg file you install with easy_install gets put in some environment variable and python knows to add it to sys.path when the python interpreter starts).

To get blah.haha to work in your example, either run easy_install blah-4.0.1-py2.7-win32.egg and then you can import haha from within python, or just put the haha module directly in site-packages.

Xanthus