Buildout and Virtualenv

Question

I am messing around with the combination of buildout and virtualenv to setup an isolated development environment in python that allows to do reproducible builds.

There is a recipe for buildout that let's you integrate virtualenv into buildout:

 tl.buildout_virtual_python

With this my buildout.cfg looks like this:

[buildout]
develop = .
parts = script
        virtualpython


[virtualpython]
recipe = tl.buildout_virtual_python
headers = true
executable-name = vp
site-packages = false

[script]
recipe = zc.recipe.egg:scripts
eggs = foo
python = virtualpython

This will deploy two executables into ./bin/:

vp
script

When I execute vp, I get an interactive, isolated python dialog, as expected (can't load any packages from the system). What I would expect now, is that if I run

./bin/script 

that the isolated python interpreter is used. But it doesn't, it's not isolated as "vp" is (meaning I can import libraries from system level). However I can run:

./bin/vp ./bin/script

Which will run the script in an isolated environment as I wished. But there must be a way to specify this to do so without chaining commands otherwise buildout only solves half of the problems I hoped :)

Thanks for your help! Patrick

Raver

Answer 1

You don't need virtualenv: buildout already provides an isolated environment, just like virtualenv.

As an example, look at files buildout generates in the bin directory. They'll have something like:

import sys
sys.path[0:0] = [
     '/some/thing1.egg',
     # and other things
     ]

So the sys.path gets completely replaced with what buildout wants to have on the path: the same isolation method as virtualenv.

Condescension

The nice thing about virtualenv is it modifies the PATH so scripts can have "#!/usr/bin/env python" just like they're supposed to and they will use the buildout version. Is it possible to do this without virtualenv? – Albumenize

virtualenv modifies the path, but that also means you should not forget to enable virtualenv every time you want to use the script. Buildout has a more permanent solution in that regard. And buildout points your script's "#!..." line at the real python your buildout was run with, so it is correct for your machine. Without virtualenv? You could write a shell script that sets the PYTHONPATH to something specific and get some of virtualenv's functionality. But not all, especially not when installing. – Condescension

Quick correction: sys.path[0:0] = [...] doesn't completely replace sys.path, it just inserts more items into the beginning of it. – Squabble

You're right. By default, buildout adds the things it wants on the path right at the front of the sys.path, but the existing path stays intact otherwise. So in that sense, virtualenv provides more isolation. Note that buildout 1.5 allows you to switch to a pure-isolation model. – Condescension

Answer 2

zc.buildout 2.0 and later does not provide the isolated environment anymore.

But virtualenv 1.9 and later provides complete isolation (including to not install setuptools).

Thus the easiest way to get a buildout in a complete controlled environment is to run the following steps (here i.e. for Python 2.7):

cd /path/to/buildout
rm ./bin/python
/path/to/virtualenv-2.7 --no-setuptools --no-site-packages --clear .
./bin/python2.7 bootstrap.py
./bin/buildout

Preconditions:

• bootstrap.py has to be a recent one matching the buildout version you are using. You'll find the latest at http://downloads.buildout.org/2/

• if there are any version pins in your buildout, ensure they do not pin buildout itself or recipes/ extensions to versions not compatible with zc.buildout 2 or later.

Fuentes

Answer 3

Had issue running buildout using bootstrap on ubuntu server, from then I use virtualenv and buildout together. Simply create virualenv and install buildout in it. This way only virtualenv has to be installed into system (in theory¹).

$ virtualenv [options_you_might_need] virtual
$ source virtual/bin/activate
$ pip install zc.buildout
$ buildout -c <buildout.cfg>

Also tell buildout to put its scripts in to virtual/bin/ directory, that way scripts appear on $PATH.

[buildout]
bin-directory = ${buildout:directory}/virtual/bin
...

---

1: In practice you probably will need to eggs what require compilation to system level that require compilation. Eggs like mysql or memcache.

Pitfall

Answer 4

I've never used that recipe before, but the first thing I would try is this:

[buildout]
develop = .
parts = script
        virtualpython


[virtualpython]
recipe = tl.buildout_virtual_python
headers = true
executable-name = vp
site-packages = false

[script]
recipe = zc.recipe.egg:scripts
eggs = foo
python = virtualpython
interpreter = vp

If that doesn't work, you can usually open up the scripts (in this case vp and script) in a text editor and see the Python paths that they're using. If you're on windows there will usually be a file called <script_name>-script.py. In this case, that would be vp-script.py and script-script.py.

Footsore

I also thought about using the "interpreter" option. If I do so, the code will break: djungle:buildout-test pbonzli$ ./bin/vp ./bin/vp: line 3: import: command not found ./bin/vp: line 5: sys.path[0:0]: command not found etc... Perhaps a bug in the script, I don't know... – Raver