What are the best practices for creating Python Distributions(eggs) on(and for) Multiple Operating Systems

Question

Ours is a python shop. We have different python packages developed inhouse and will be deployed onto customers' environments(machines).

This is how our development and release cycle happens.

Once developers complete "testing" of a package, a distribution(egg file) of the package is prepared and pushed to a central archiving place. WHen we want to deploy our software to Customers, the same distributions(egg files) will be downloaded and installed in their environment.

Assuming the "testing" happens on multiple operating systems(to check the compatibility of the API across platforms), what is the best practice to prepare distributions and be pushed to the central archiving place.

Is it best to have operating system specific eggs on the archiving server(like, samplepkg-1.0.0.win32.egg and samplepkg-1.0.0.linux.egg ? Not sure how they can be prepared in this way using setuptools. ) Or Have a single egg because API remains same across platforms ? Any other practice which is followed by the community ?

Nutty

Answer 1

You can use a single package if:

1. The package does not use functions/classes that are not available on all your target platforms (see e.g. chapters 36-39 of the Python standard library reference for version 2.7.2 for stuff that you shouldn't use in that case)
2. You are not using extensions written in C/C++ that need to be compiled for every platform.

It it generally a good idea to stay away from OS specific functions that are not available on all your target platforms. The standard library is quite well documented in that respect.

Auctorial

Answer 2

I think that in this case, using a single package would be more complicated due to the reasons that Roland mentioned above. In your development environment you could have separate folders for separate platforms, each with the platform specific code (such as extensions/libraries written in C/C++). You would need to mimic your setuptools within these folders to produce separate eggs, but it would ultimately be less complex (I think, without knowing more about what code you are actually working with) than trying to put everything into one package.

In either case, reading the documentation on the standard library, as well as distutils (http://docs.python.org/distutils/) should help you find your solution.

Scapegrace

Answer 3

Platform-specific eggs are only intended for distributing packages containing C code; otherwise the egg files themselves are platform-independent and you only need to distribute one, platform-independent egg.

If you are using automated installation tools or pkg_resources' runtime APIs for finding libraries and plugins, you can actually just dump all the eggs in a single directory, and the installation tool or runtime API will pick which egg to install or import from.

tl;dr version: the way setuptools builds eggs is the way they should be distributed; if you try to make a cross-platform egg into a platform-dependent one or vice versa, you're likely to experience some pain. ;-)

Mccartney

Answer 4

If you only have python modules, keep all the difference hidden in python, either same files (python std lib) or separate files (pyserial).

If you have compiled modules, it's a bit trickier.

I use the following directory structure for a project that needs compiled extension:

./lib/display.py  # frontend, platform-independent
./lib.linux-x86_64-2.6/_display.so
./lib.linux-armv5tejl-2.6/_display.so

And this code in the beginning of the beginning of the program:

sys.path.append("lib.%s-%s-%s.%s" % ((posix.uname()[0].lower(),
                                      posix.uname()[4])
                                     +sys.version_info[:2]))

You can wrap structure like this in an .egg too, as long as you specify that it's not zip-safe.

Appanage