How to maintain long-lived python projects w.r.t. dependencies and python versions?

Question

short version: how can I get rid of the multiple-versions-of-python nightmare ?

long version: over the years, I've used several versions of python, and what is worse, several extensions to python (e.g. pygame, pylab, wxPython...). Each time it was on a different setup, with different OSes, sometimes different architectures (like my old PowerPC mac).

Nowadays I'm using a mac (OSX 10.6 on x86-64) and it's a dependency nightmare each time I want to revive script older than a few months. Python itself already comes in three different flavours in /usr/bin (2.5, 2.6, 3.1), but I had to install 2.4 from macports for pygame, something else (cannot remember what) forced me to install all three others from macports as well, so at the end of the day I'm the happy owner of seven (!) instances of python on my system.

But that's not the problem, the problem is, none of them has the right (i.e. same set of) libraries installed, some of them are 32bits, some 64bits, and now I'm pretty much lost.

For example right now I'm trying to run a three-year-old script (not written by me) which used to use matplotlib/numpy to draw a real-time plot within a rectangle of a wxwidgets window. But I'm failing miserably: py26-wxpython from macports won't install, stock python has wxwidgets included but also has some conflict between 32 bits and 64 bits, and it doesn't have numpy... what a mess !

Obviously, I'm doing things the wrong way. How do you usally cope with all that chaos ?

Ducal

pygame supports 2.6, actually. If macports forced you to install 2.4 because of that, macports is wrong. – Penstock

oop,s you're right. I suppose I was mistaken a-long-time-ago by the confusing naming schemes of macports (i.e. py26-game VS just py-game). Now I shall be more cautious :-) But still... – Ducal

Thanks for your answers everyone. But my problem is not quite solved here: I am on an OSX 10.6 64-bit machine, and I want to write a wxpython+matplotlib program. How do I do ? – Ducal

Did you follow my advice and install the Enthought Python Distribution? If you do so, all the components will be installed at once: numpy, matplotlib, wxpython and so on. – Mohl

@Olivier: I did consider it, but then I had a look at the price tag. I can't afford $200 just for this project. – Ducal

Answer 1

Some tips:

• on Mac OS X, use only the python installation in /Library/Frameworks/Python.framework.
• whenever you use numpy/scipy/matplotlib, install the enthought python distribution
• use virtualenv and virtualenvwrapper to keep those "system" installations pristine; ideally use one virtual environment per project, so each project's dependencies are fulfilled. And, yes, that means potentially a lot of code will be replicated in the various virtual envs.

That seems like a bigger mess indeed, but at least things work that way. Basically, if one of the projects works in a virtualenv, it will keep working no matter what upgrades you perform, since you never change the "system" installs.

Mohl

Answer 2

I solve this using virtualenv. I sympathise with wanting to avoid further layers of nightmare abstraction, but virtualenv is actually amazingly clean and simple to use. You literally do this (command line, Linux):

virtualenv my_env

This creates a new python binary and library location, and symlinks to your existing system libraries by default. Then, to switch paths to use the new environment, you do this:

source my_env/bin/activate

That's it. Now if you install modules (e.g. with easy_install), they get installed to the lib directory of the my_env directory. They don't interfere with existing libraries, you don't get weird conflicts, stuff doesn't stop working in your old environment. They're completely isolated.

To exit the environment, just do

deactivate

If you decide you made a mistake with an installation, or you don't want that environment anymore, just delete the directory:

rm -rf my_env

And you're done. It's really that simple.

virtualenv is great. ;)

Jesusitajet

Answer 3

Take a look at virtualenv.

Izy

yay, and here we go for one more layer of nightmare :-D – Ducal

@Gyom: Blasphemy! Every computer science problem is solvable by Just One More Abstraction Layer! – Ayesha

Answer 4

What I usually do is trying to (progressively) keep up with the Python versions as they come along (and once all of the external dependencies have correct versions available).

Most of the time the Python code itself can be transferred as-is with only minor needed modifications.

My biggest Python project @ work (15.000+ LOC) is now on Python 2.6 a few months (upgrading everything from Python 2.5 did take most of a day due to installing / checking 10+ dependencies...)

In general I think this is the best strategy with most of the interdependent components in the free software stack (think the dependencies in the linux software repositories): keep your versions (semi)-current (or at least: progressing at the same pace).

Affective

sure, but keeping one big project up-to-date is quite different than maintaining plenty of small projects. and some programs I'm working with are stable and third-party, so it's simpler just to shebang these against the right version of python rather than constantly updating them. – Ducal

Answer 5

• install the python versions you need, better if from sources
• when you write a script, include the full python version into it (such as #!/usr/local/bin/python2.6)

I can't see what could go wrong.

If something does, it's probably macports fault anyway, not yours (one of the reasons I don't use macports anymore).

I know I'm probably missing something and this will get downvoted, but please leave at least a little comment in that case, thanks :)

Whitley

what could go wrong ? for example the fact that some dependencies (here, wxpython) won't compile on my machine. Of course, you can blame macports instead of contributing to it, but that does not solve my problem :-) – Ducal

Answer 6

I use the MacPorts version for everything, but as you note a lot of the default versions are bizarrely old. For example vim omnicomplete in Snow Leopard has python25 as a dependency. A lot of python related ports have old dependencies but you can usually flag the newer version at build time, for example port install vim +python26 instead of port install vim +python. Do a dry run before installing anything to see if you are pulling, for example, the whole of python24 when it isn't necessary. Check portfiles often because the naming convention as Darwin ports was getting off the ground left something to be desired. In practice I just leave everything in the default /opt... folders of MacPorts, including a copy of the entire framework with duplicates of PyObjC, etc., and just stick with one version at a time, retaining the option to return to the system default if things break unexpectedly. Which is all perhaps a bit too much work to avoid using virtualenv, which I've been meaning to get around to using.

Gloze

I left MacPorts a long time ago. Just use pip and virtualenv for your Python needs and binary files for everything else. Ubuntu is the solution if you want proper package management. – Agle

Yeah, if you can get away with using the system version of Python (or a nicely packaged alternative, like Enthought's one) definitely do so. MacPorts is just OK in general but completely falls down when dealing with scripting languages. – Istle

Answer 7

I've had good luck using Buildout. You set up a list of which eggs and which versions you want. Buildout then downloads and installs private versions of each for you. It makes a private "python" binary with all the eggs already installed. A local "nosetests" makes things easy to debug. You can extend the build with your own functions.

On the down side, Buildout can be quite mysterious. Do "buildout -vvvv" for a while to see exactly what it's doing and why.

http://www.buildout.org/docs/tutorial.html

Oma

Answer 8

At least under Linux, multiple pythons can co-exist fairly happily. I use Python 2.6 on a CentOS system that needs Python 2.4 to be the default for various system things. I simply compiled and installed python 2.6 into a separate directory tree (and added the appropriate bin directory to my path) which was fairly painless. It's then invoked by typing "python2.6".

Once you have separate pythons up and running, installing libraries for a specific version is straightforward. If you invoke the setup.py script with the python you want, it will be installed in directories appropriate to that python, and scripts will be installed in the same directory as the python executable itself and will automagically use the correct python when invoked.

I also try to avoid using too many libraries. When I only need one or two functions from a library (eg scipy), I'll often see if I can just copy them to my own project.

Countermine

thanks for your asnwers. however, the problem I describe in the question precisely occurs because of libraries. And I can't "copy" these libraries into the project as they are written in C (scipy, matplotlib, wxpython) and not in pure python. – Ducal