pip: Why sometimes installed as egg, sometimes installed as files

Question

Where can you force pip to install as "flat" and not as "egg".

For me it seems random. Sometimes it gets installed as egg, sometime as flat.

pip help install shows only an option --egg which forces an egg installation. But I could not find a --flat option.

The packages are from an own pypiserver, and uploaded like this:

python setup.py sdist upload -r internal

Output of pip during install:

Best match: foo-client 2015.2
Downloading https://installserver:40443/pypi/packages/foo_client-2015.2.tar.gz
Processing foo_client-2015.2.tar.gz
Writing /home/bar_eins_daad/tmp/easy_install-z20B7b/foo_client-2015.2/setup.cfg
Running foo_client-2015.2/setup.py -q bdist_egg --dist-dir /home/bar_eins_daad/tmp/easy_install-z20B7b/foo_client-2015.2/egg-dist-tmp-GO1snX

I don't know why bdist_egg gets used here. Does it force creating an egg install?

The setup.py does use setuptools not distutils.

The package on our pypiserver looks like this:

tar -tzf packages/foo_client-2015.3.tar.gz

content:

foo_client-2015.2/
foo_client-2015.2/foo_client.egg-info/
foo_client-2015.2/foo_client.egg-info/SOURCES.txt
foo_client-2015.2/foo_client.egg-info/top_level.txt
foo_client-2015.2/foo_client.egg-info/dependency_links.txt
foo_client-2015.2/foo_client.egg-info/PKG-INFO
foo_client-2015.2/setup.cfg
foo_client-2015.2/PKG-INFO
foo_client-2015.2/foo_client/
foo_client-2015.2/foo_client/models.py
...

Background

pip seems to sometimes install packages several times if zipped eggs are installed.

Update

I found under which condition the package gets installed as egg: if it gets installed via python setup.py develop (it is an install_requires dependency).

If I use pip install foo_client it gets installed flat (the way I want it).

Update2

The very ugly part: If the egg gets installed, the old version installed flat does not get removed.

Version: pip 1.5.6

Leslee

Can you elaborate? The --egg option is there to make pip install an egg (binary) distribution if available, rather than to install from source (the default), for example. Can you give us some sample packages that show the difference? – Isis

@MartijnPieters sorry, our pypiserver is not available for the public. Is there an other way to give you more details? – Leslee

Are there no publicly available packages that exhibit this behaviour at all? – Isis

@MartijnPieters we mostly install from our own pypiserver. Even packages from pypi.org. We first download to our own pypiserver, then we install from it. – Leslee

Then surely you can find a package from among those that illustrate the behaviour you are talking about. This has nothing to do with what server served the package. – Isis

@MartijnPieters I can't find a package which installs as egg but should not. I updated some infos about our python packages. – Leslee

Pip does install the egg metadata. That is normal behaviour. The package is still going to end up unzipped in your site-packages. Are you saying you have zipped egg files there, or are there only .egg-info directories there? – Isis

install_requires indeed installs the requirement as an egg. It won't remove other versions because what is installed now is only a installation dependency. You can remove those again after installation has completed, in any case. – Isis

@MartijnPieters íf not install_requires what should be used in setup.py? I read these docs: python-packaging-user-guide.readthedocs.org/en/latest/… – Leslee

I don't know what problem you are trying to solve there. – Isis

@MartijnPieters I want to have python packages installed flat and not as egg. It should not matter how the package gets installed (via pip directly or via python setup develop). – Leslee

I was talking about the install_requires field. Why are you using it? – Isis

@MartijnPieters I want to define a dependency. Example: package foo_client_for_customer_x needs package foo_client. Like documented here: packaging.python.org/en/latest/…. I see no other use case. Why do you ask? – Leslee

My mistake, my apologies. I was getting mixed up with setup_requires. install_requires should not result in eggs being installed, pip will install those correctly still. – Isis

Answer 1

This does not solve the question why I get sometimes zipped eggs, and sometimes not. But it helps.

You can use this in your ~/.distutils.cfg to avoid installation of zipped eggs:

[easy_install]
zip_ok = False

Leslee

Answer 2

If you are the author of the package, you can use the flag zip_safe=False in setup.py.

setup(
    name = "HelloWorld",
    ...
    zip_safe = False,
)

If you are a user who wants to improve the package, you can install it by pip install -e foo_package. The option -e or --editable installs a project in an editable mode (i.e. setuptools "develop mode"), not zipped. It creates a link from source into site-packages and compiles .../bin scripts, but it doesn't copy the source into "site-packages". Those packages can not be updated automatically. It is the main reason why it is not intended as a usual way of installing packages, but only for those that need to be customized or fixed.

Edit: Django is a typical framework that requires zip_safe=False for its applications, because they are not a pure Python, but they contain also templates with html, css, i18n resources etc. Is your question related to Django?

Balefire

Sad, I am not the author of the package. I found a solution now. See #27965460 – Leslee

This is actually pretty useful to control the creations of *.egg folders when you write your own package. Thank you! – Cutch

thanks, I just fixed a django package :-D – Yester

Answer 3

I was having this egg-only install problem, and it turned out I had failed to git add the __init__.py in the root of my package. It was driving me crazy that this would work:

pip install .

...but this would fail:

mkdir /tmp/piptest
cd /tmp/piptest
git clone $OLDPWD .
pip install .

It was hard to notice the difference using diff -r . $OLDPWD because there are so many non-committed pyc files and development tool scripts.

This is probably not the answer for this OP, but I hope it helps someone who Googles "pip only installing the egg" as I did.

Trilbie

Adding a subfolder for my package (and an __init__.py inside, necessary?) was the answer for me. Thanks – Shoal

I'm glad it helped you, @BrianPeterson. But honestly, I put answers like this on the questions that Google leads me to because at some point years from now, I'll need it again. – Trilbie

Answer 4

Some docs say:

For maximum performance, Python packages are best installed as zip files.

and

You can pass a True or False value for the zip_safe argument to the setup() function, or you can omit it. If you omit it, the bdist_egg command will analyze your project’s contents to see if it can detect any conditions that would prevent it from working in a zipfile.

So it's probably fine unless... it's not fine in your case. My project tried to read files and failed due to the zipping.

Notably, this only happened with python setup.py install, not pip install . @guettli's fix worked fine, but I put it in setup.cfg:

[easy_install]
zip_ok = False

Gregg

Answer 5

I had the same issue as @guettli and solved it by unzipping and extracting the archive first then running:

pip -e install /srv/mypkg-1.1.0

where /srv/mypkg-1.1.0/ is the top level directory of project/package that has a setup.py file in it.

mypkg-1.1.0 was installed in site-packages and mypkg.py was listed in [virtualenv]/bin

note: '-e' flag is optional.

Thanks

Insatiate

Since posting I've learned 'pip install' with '-e', editable option; installs the package in 'devloper mode'. A /srv/mypkg-1.1.0/mypkg.egg-info directory is created in the project directory. The source code can be edited without building the package each time. Not recommended for a production install. Therefore as per my original post, I'm unsure whether installing from tarball or from flat file makes the difference as the first time I installed it my tarball was in /tmp and for all I know the egg-info directory may have created there. I'll have to test to confirm. – Insatiate