Python and OpenMP C Extensions

Question

I have a C extension in which I'd like to use OpenMP. When I import my module, though, I get an import error:

ImportError: /home/.../_entropysplit.so: undefined symbol: GOMP_parallel_end

I've compiled the module with -fopenmp and -lgomp. Is this because my Python installation wasn't compiled with the -fopenmp flag? Will I have to build Python from source? Or is there some other possibility? This is the only time I actually use openmp in my module:

unsigned int feature_index;
#pragma omp parallel for
for (feature_index = 0; feature_index < num_features; feature_index++) {

I'd like to stick with openmp if it's possible, just because it's so easy and the parallelization in this case suits it well.

EDIT: I bit the bullet and recompiled Python with OpenMP support. My module works perfectly now, but this isn't really a great solution. I can't really distribute this if it requires a complete recompile of Python. So does anybody know some way around this? Would ctypes work, maybe?

SOLVED! It was a simple linking issue. (I rebuilt Python for that?!) OpenMP wasn't being properly linked during the compilation of the module. So it IS possible to load a C Python extension that uses OpenMP.

Md

You might consider copying your solution into a "real" answer posted under this question, so that it's easier to see (and can be upvoted). – Lib

Answer 1

Just to make it clearer, here is what your setup.py should look like:

ext = Extension(
      'milk.unsupervised._som',
      sources = ['milk/unsupervised/_som.cpp'],
      extra_compile_args=['-fopenmp'],
      extra_link_args=['-lgomp'])


...
setup(..., ext_modules = [ext])

Cephalonia

Now, how to do that in a cross-platform way that works with versions of gcc, msvc and Clang that support openmp and fallsback gracefully otherwise? – Spirt

@ColonelPanic: I would also love to know that. – Cephalonia

Answer 2

I know this is a dated post, but I'll share my experience as I too ran into this exact same issue, but when using f2py at the command line. I was originally compiling my OpenMP enabled Fortran 90 subroutine using

f2py --fcompiler=gfortran --f90flags='-fopenmp -lgomp' -m sub -c sub.90

which succesfully created the shared object sub.so. However, trying to import this from a Python shell produced the similar undefined symbol ImportError. However, as the original author stated it's because I was trying to pass both -fopenmp and -lgomp to the compiler, whereas only -fopenmp should be passed to it, and -lgomp should be passed to the linker.

Therefore, I should have been doing the following

f2py --fcompiler=gfortran --f90flags='-fopenmp' -lgomp -m sub -c sub.f90

And that's it, problem solved, I can now import my subroutine.

Haldane

Answer 3

It was a simple linking issue. OpenMP wasn't being properly linked during the compilation of the module. So it IS possible to load a C Python extension that uses OpenMP. -fopenmp has to be passed to the compiler and -lgomp to the linker -- if you're using distutils, make sure your setup.py is configured properly. Rebuilding Python also worked, I'm guessing, because I had properly linked OpenMP with Python, so when Python loaded the module the library was already properly linked to.

Md