How may I override the compiler (GCC) flags that setup.py uses by default?

Question

I understand that setup.py uses the same CFLAGS that were used to build Python. I have a single C extension of ours that is segfaulting. I need to build it without -O2 because -O2 is optimizing out some values and code so that the core files are not sufficient to pin down the problem.

I just need to modify setup.py so that -O2 is not used.

I've read distutils documentation, in particular distutils.ccompiler and distutils.unixccompiler and see how to add flags and libs and includes, but not how to modify the default GCC flags.

Specifically, this is for a legacy product on Python 2.5.1 with a bunch of backports (Fedora 8, yes, I know...). No, I cannot change the OS or Python version and I cannot, without great problems, recompile Python. I just need to build a one off of the C extension for one customer whose environment is the only one segfaulting.

Demmer

Answer 1

• Prepend CFLAGS="-O0" before you run setup.py:

  % CFLAGS="-O0" python ./setup.py
  

  The -O0 will be appended to CFLAGS while compiling, therefore will override previous -O2 setting.

• Another way is add -O0 to extra_compile_args in setup.py:

  moduleA = Extension('moduleA', .....,
          include_dirs = ['/usr/include', '/usr/local/include'], 
          extra_compile_args = ["-O0"], 
          )
  

• If you want to remove all default flags, use:

  % OPT="" python ./setup.py
  

Chanel

when i run with the above setting, it shows both CFlaGS, -O2 as well as -O0. it is not overriding it – Alsup

@Alsup For compilers such as gcc and clang, the later flag takes precedence, so if -O0 appears on the command line after -O2, then -O0 is overriding it. – Lois

Note that compiler/linker environment variables are ignored on Windows. – Jourdain

OPT doesn't work. – Sphygmomanometer

Are you sure @EvgenKo423? – Manama

@matanster Yeah, I tried it myself and the link above shows how it really works. Note that I'm talking about environment variables only. I mean they are ignored by Python that is, but the compiler itself will of course use its variables if you'll set them. You just can't use CFLAGS to modify MSVC flags on Windows. – Jourdain

Answer 2

I ran into this problem when I needed to fully remove a flag (-pipe) so I could compile SciPy on a low-memory system. I found that, as a hack, I could remove unwanted flags by editing /usr/lib/pythonN.N/_sysconfigdata.py to remove every instance of that flag, where N.N is your Python version. There are a lot of duplicates, and I'm not sure which are actually used by setup.py.

Litotes

This did the trick as a work around...Just had to edit the CFLAGS entry – Schiedam

Answer 3

distutils/​setuptools allows any compiler/​linker flags to be specified with extra_compile_args/​extra_link_args argument when defining a Python extension in setup.py script. These extra flags will be added after default ones and will override any mutually exclusive flags present earlier.

For regular use, however, this is not much useful as a package you distribute through PyPI can be built by different compilers having an incompatible options.
The following code allows you to specify these options in extension- and compiler-specific way:

from setuptools import setup
from setuptools.command.build_ext import build_ext


class build_ext_ex(build_ext):

    extra_compile_args = {
        'extension_name': {
            'unix': ['-O0'],
            'msvc': ['/Od']
        }
    }

    def build_extension(self, ext):
        extra_args = self.extra_compile_args.get(ext.name)
        if extra_args is not None:
            ctype = self.compiler.compiler_type
            ext.extra_compile_args = extra_args.get(ctype, [])

        build_ext.build_extension(self, ext)


setup(
    ...
    cmdclass = {'build_ext': build_ext_ex},
    ...
)

Of course you could simplify it if you want all extensions to use the same (but still compiler-specific) options.

Here is a list of supported compiler types (as returned by setup.py build_ext --help-compiler):

--compiler=bcpp     Borland C++ Compiler
--compiler=cygwin   Cygwin port of GNU C Compiler for Win32
--compiler=mingw32  Mingw32 port of GNU C Compiler for Win32
--compiler=msvc     Microsoft Visual C++
--compiler=unix     standard UNIX-style compiler

Jourdain

It's even compatible with Python 2.5. =)) – Jourdain

Could you possibly lay down the compatibility aspect or concerns that you have mentioned in a little more depth or breadh? – Manama

@matanster Your extensions could be built on Windows with MSVC or on Unix-likes with GCC. These compilers have different options, but extra_compile_args only allows you to specify a single set of options, so you have to use the code above or something else to overcome this inconvenience. – Jourdain