How to tell distutils to use gcc?

Question

I want to wrap a test project containing C++ and OpenMP code with Cython, and build it with distutils via a setup.py file. The content of my file looks like this:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext


modules = [Extension("Interface",
                     ["Interface.pyx", "Parallel.cpp"],
                     language = "c++",
                     extra_compile_args=["-fopenmp"],
                     extra_link_args=["-fopenmp"])]

for e in modules:
    e.cython_directives = {"embedsignature" : True}

setup(name="Interface",
     cmdclass={"build_ext": build_ext},
     ext_modules=modules)

The -fopenmp flag is used with gcc to compile and link against OpenMP. However, if I just invoke

cls ~/workspace/CythonOpenMP/src $ python3 setup.py build

this flag is not recognized, because the compiler is clang:

running build
running build_ext
skipping 'Interface.cpp' Cython extension (up-to-date)
building 'Interface' extension
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Interface.cpp -o build/temp.macosx-10.8-x86_64-3.3/Interface.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Parallel.cpp -o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
Parallel.cpp:24:10: warning: unknown pragma ignored [-Wunknown-pragmas]
        #pragma omp parallel for
                ^
1 warning generated.
c++ -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-3.3/Interface.o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -o build/lib.macosx-10.8-x86_64-3.3/Interface.so -fopenmp
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1

I've unsucessfully tried to specify gcc:

cls ~/workspace/CythonOpenMP/src $ python3 setup.py build --compiler=g++-4.7
running build
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with 'g++-4.7' compiler

How can I tell distutils to use gcc?

Laughable

Answer 1

Try setting the "CC" environment variable from inside the setup.py with os.environ.

Obeded

os.environ["CC"] = "g++-4.7" os.environ["CXX"] = "g++-4.7" just worked – Laughable

This does not work for me. My problem is that setup.py wants to use icc. When I set CC=gcc, it does try to use gcc, but it keeps using the command line arguments appropriate for icc, among them -fp-model strict which gcc doesn't understand, and aborts. So it looks like just setting CC is not the right way to do this. – Engage

Distutils does not check for CXX, so setting that may end up confusing you. You might want to remove the suggestion to add "CXX" since it is misleading. – Megrims

psutil.py was partly ignoring the CC setting so I found a really ugly way to build it on Solaris 11: ln -s /usr/bin/gcc /usr/bin/cc. – Advise

I also found that the CC setting was partially ignored – Cockahoop

Answer 2

I just took a look at the distutils source, and the --compiler option expects "unix", "msvc", "cygwin", "mingw32", "bcpp", or "emx". It checks the compiler name you want by checking the CC environment variable. Try calling build like this:

CC=gcc python setup.py build

You don't need to set CXX, it doesn't check for that.

Megrims

On ubuntu 12.04 it really doesn't look at the CXX environmental variable, that is not what one would expect. (might be the case that it doesnt check on other platforms either ofcourse) – Venessavenetia

@Venessavenetia I'm having a hard time understanding your comment. Are you agreeing or disagreeing with this answer? – Megrims

Well in build systems like the autotools, CMake (which generate Makefiles) and Makefiles. They inspect the CC and CXX environmental variables. Then in the makefile they execute $(CC) -c csource.c or $(CXX) -c cppsource.cpp to create objectfiles. And apparently distunitils break with this conventions. This is why I upvoted your answer because the phrase "you don't need to set CXX" solved my problem, but many people using those mentioned build tools will expect that CC is for building C and CXX for building C++ programs. – Venessavenetia

I see what you were trying to say. Yes, it is confusing indeed. I'm not sure the original authors were thinking things through in general with distutils. – Megrims

in macOS I also had to set CXX=g++-9 in order to get it to work. Otherwise it was using clang++. – Bonham

Answer 3

Just in case some others are facing the same problem under Windows (where CC environment variable wouldn't have any effect) :

• Create file "C:\Python27\Lib\distutils\distutils.cfg" and write this inside :

Code :

[build]
compiler = mingw32

• Remove all instances of "-mno-cygwin" gcc option from file "C:\Python27\Lib\distutils\cygwinccompiler.py" :

This :

    self.set_executables(compiler='gcc -mno-cygwin -O -Wall',
                         compiler_so='gcc -mno-cygwin -mdll -O -Wall',
                         compiler_cxx='g++ -mno-cygwin -O -Wall',
                         linker_exe='gcc -mno-cygwin',
                         linker_so='%s -mno-cygwin %s %s'
                                    % (self.linker_dll, shared_option,
                                       entry_point))

Becomes this :

self.set_executables(compiler='gcc -O -Wall',
                     compiler_so='gcc -mdll -O -Wall',
                     compiler_cxx='g++ -O -Wall',
                     linker_exe='gcc',
                     linker_so='%s %s %s'
                                % (self.linker_dll, shared_option,
                                   entry_point))

The second point can be necessary in case you are using a recent version of gcc, where the deprecated option -mno-cygwin has been removed.

Hope this will help even if it is not directly related to the OP real needs (but still related to the question's title...)

Yawata

I was actually struggling with this problem a few minutes ago but I am now clear Thanks man! and why are we adding a new file to the distutils? – Firefly

It's 2014 now, and I have GCC version 4.3.3 and Python verison 3.4. I've implemented your solution with no results. I get the same error. I wrote [build] compiler = mingw32 into the config file, and deleted -mcygwin ... – Mercuri

My answer might be outdated. That is completely possible. I am sorry if this is the case, but it's been a long time I haven't been working on portability stuffs under Windows (strict "Linux work" for months), and actually won't be able to help you with this, right now... Make sure you created distutils.cfg at the right place (the requirements might have changed for this file). Have you tried to compile using directly python3 setup.py build --compiler=mingw32? – Yawata

Answer 4

According to this wiki, Python versions after 3.4 do not support MinGW anymore. CPython 3.7 for Windows is compiled with MSC v.1916. When I try to use above-mentioned method with distutils.cfg, I then get an error from distutils: Unknown MS Compiler Version 1916. Looks like it has a hardcoded table of msvcr libraries in its cygwincompiler.py file (which is also responsible for MinGW), and last version known to that file is 1600 from VS2010 / MSVC 10.0.

Bucephalus

did you find a key to your problem? – Footer

Answer 5

Try this: http://mail.python.org/pipermail/distutils-sig/2002-August/002944.html

In short, it appears that you should try: python setup.py build --compiler=g++ first.

Apterygial

Found this too, but python setup.py build --compiler=g++ gives the same result as above. – Laughable

what if you don't specify the version? – Apterygial

python setup.py build --compiler=g++ vs. python setup.py build --compiler=g++-4.7 – Apterygial

Answer 6

On linux while using distutils.ccompiler do os.environ('CC')='gcc' and then call distutils.sysconfig.customize_compiler(compiler)

It will do the job.

Patellate