Buildout with part build with Cython

Question

I'm facing problem with cython in buildout.

One of the part is a module build with cython from a .c file and a .pyx file.

I've already try many solutions :

• Sean Gillies Blog / 814 / Adding pyproj to a buildout
• gefira blog » Getting rid of ‘No Cython, trying Pyrex…’ in zc.buildout

But all ends with error :

ImportError: No module named Cython.Distutils

This append when buildout load the setup.py of this cython module.

Note that the setup.py is functional when called from its own directory and with an interpreter in buildout bin directory.

Thanks for your help.

Linnell

Are you sure Cython is properly installed and you've set the PYTHONPATH environment variable to include the dir where the Cython modules live? – Rem

Did you mean at the system level ? This will make this buildout not possible to be deployed on a system without cython ... ? – Tape

What do you mean by "system level"? – Rem

I mean installed in the site-package. Thanks to looking at my questions. – Tape

Well, you need to have it installed somewhere, either in /usr/local/lib/python2.X/site-packages or in your homedir. In the latter case, PYTHONPATH must be set properly. – Rem

Answer 1

The articles you linked tell you how to install Cython from a buildout, but to do that you need write access to the site-packages folder. There are 3 ways you should be able to run those buildouts:

1. Run it as root and install Cython into the system python's site-packages. Usually you want to avoid doing this.

2. Compile your own Python. This may be the only option if you want to use a version of Python that is not already on the system. There are buildout configurations that will let you build any version of Python from inside the buildout.

3. Use virtualenv. This will create (in the buildout folder) a complete virtual environment for Python including your own site-packages folder. This is usually the best way to run a buildout that can use the system Python (or any other Python that you have already installed system wide).

I suggest you make using virtualenv part of the installation instructions for your software.

Letter

Thank you for this proposition, i will dig this way. – Tape

Answer 2

I found today this recipe https://pypi.python.org/pypi/mr.cython/1.0 which solve the problem.

This solve the problem by installing cython with an extention recipe so it is available when buildout run setup.py develop

The problem was to build a cython module without having cython installed as system level.

Linnell

The answer looks fine to me. OP is answering to his own question with a relevant solution and context. – Butt

The fact that OP comes back several years after the original question to supply an answer is very much in the spirit of StackOverflow and deserves upvotes. See relevant XKCD: xkcd.com/979 – Butt

@HåkenLid: Well said, I upvoted, too. @ OP: You might also accept your answer :-) – Sandblast

Answer 3

There is no way to do this as a single step, but it's easy to do as a two step process.

Use:

python bootstrap.py
./bin/buildout install cython
./bin/cpy bootstrap.py
./bin/cpy ./bin/buildout

The reason this is possible is because buildout supports an obscure option 'install' which no one ever talks about, but you can use it, like this:

[buildout]
parts = deps py   # <---- Notice we don't depend on cython here
eggs =
  whatever
  kdist
  nark
  kivy # <--- But we do have a module that requires cython to build
develop =
  .
  lib/nark
  lib/kivy-dist

[cython] # <---- By calling ./bin/buildout install cython we trigger this
recipe = zc.recipe.egg:script
parts = cython-py
interpreter = cpy # <--- Which makes our dummy ./bin/cpy 
eggs =
  cython
  pyinstaller

[deps]
recipe = zc.recipe.egg:eggs
eggs = ${buildout:eggs}

[py]
recipe = zc.recipe.egg:script
interpreter = py
eggs = ${buildout:eggs}

The cute thing about this approach is that running buildout a second time clears out the bin directory so at the end of the day, you're left with a bin directory that looks like this:

$ ls bin/
buildout garden   py

No leftover packages that may or may not hang around in your virtualenv and screw things up later. That why we're using buildout in the first place right?

...of course, if you want cython to hang around, juts stick it into the dependencies at the top as well.

Impatiens

So cool didn't know about about that parameter and it's the missing piece I was looking for (to not rebuild all parts always when running buildout). Gracias! – Sandblast