How do you compile Python C/C++ extensions for different OS/versions of Python?

Question

I have noticed that several mature Python libraries have precompiled versions for most architectures (Win32/Win-amd64/MacOS) and versions of Python. What is the standard way to cross-compile your extensions for different environments? Wine? Virtual machines? Crowd sourcing?

Gatecrasher

You can find some limited information about this in the official Python docs. – Coast

Thanks, but this answers the question only partially. – Gatecrasher

I think Christoph Gohlke could answer this question. I brought his attention to it. – Cupulate

Is this a question of how to develop cross platform code or the actual cross compiling? – Augusto

Sorry for the late answer. The question is about actual cross compiling. I have seen that some serious projects (i.e. Numpy) have a package for almost every popular architecture. So the question is how do they usually do this? Do people have a physical/virtual machine with these OSes, or they crowd source actual compilation, or there is some online service for cross-compiling? – Gatecrasher

Answer 1

We use Virtual Machines and a Hudson server.

We have a Virtual Machine for each architecture we support (generally compiling doesn't stretch the resources allocated to them, so a VM is fine). I guess the configuration of each VM could be managed by something like Puppet or Chef to ensure it is a suitable build environment. You could of course use real machines if you have the hardware, but you'd want to avoid using machines which people are actually using (e.g. developer workstations).

We then use a multi-configuration project in Hudson to setup a build matrix. The build matrix allows us to (with a single click) build multiple python versions, on multiple architectures, but in theory you can build every and any combination you can setup in your matrix. Of course you could use Jenkins instead.

Rowdyish

guess the vm way might be "semi" illegal if one of your targets is osX :) still seems like the VM way is probably the best way to go. – Mullock

OSX has allowed virtualization for a little while now: macworld.com/article/1163755/… – Nazar

We actually use a little mac mini farm specifically for OSX rather than virtualised servers - sorry, should have clarified that in the original answer. Especially when buying from ebay, these are cheap enough for what we require, and are just stacked in a corner of the machine room. But as hbar says, virtualising these is also a legal option now. – Rowdyish

Answer 2

SWIG provides a path for multiplatform code generation.

Augusto

Sorry, I was not clear about it. The question is about actual compiling, not the code development. – Gatecrasher

Answer 3

All of my Python extension modules are C++, not C so I use boost Python. I also use virtual machines when I need to support different operating systems. Boost's bjam build driver allows you to build with different versions of Python (2.6, 2.7) different versions of g++ and various other things. If I had an extension module that was very popular and many people wanted to use it on platforms that I do not have, then I would just make sure my code was very portable (it should be anyway) and provide instructions on how I build it with bjam using several different examples for different Python versions, etc. That would be enough to get them started. If this works, you could ask them to contribute their builds back so others could use them (unsupported by you of course).

Come