How can I build my C extensions with MinGW-w64 in Python?

Question

So I have a few Python C extensions I have previously built for and used in 32 bit Python running in Win7. I have now however switched to 64 bit Python, and I am having issues building the C extension with MinGW-w64.

I made the changes to distutils as per this post, but I am getting some weird errors suggesting something is wrong:

$ python setup.py build
running build
running build_ext
building 'MyLib' extension
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -mdll -O -Wall -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c MyLib.c -o build\temp.win-amd64-2.7\Release\mylib.o
MyLib.c: In function 'initMyLib':
MyLib.c:631:5: warning: implicit declaration of function 'Py_InitModule4_64' [-Wimplicit-function-declaration]
writing build\temp.win-amd64-2.7\Release\MyLib.def
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -shared -s build\temp.win-amd64-2.7\Release\mylib.o build\temp.win-amd64-2.7\Release\MyLib.def -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -lpython27 -o build\lib.win-amd64-2.7\MyLib.pyd
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x13d): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1275): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1eef): undefined reference to `__imp_PyExc_ImportError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f38): undefined reference to `__imp_PyExc_AttributeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f4d): undefined reference to `__imp_PyCObject_Type'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f61): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1fc7): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1ffe): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x2042): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x206c): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x208a): more undefined references to `__imp_PyExc_RuntimeError' follow
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x20a7): undefined reference to `__imp_PyExc_ImportError'
collect2.exe: error: ld returned 1 exit status
error: command 'x86_64-w64-mingw32-gcc' failed with exit status 1

I have googled around quite a bit to find information, but it's not easy to find a definite answer. Could someone shed some light on this? What further changes should I do to be able to successfully build C extensions for 64 bit Python in Win7?

EDIT:

After some helpful pointers in cgohlke's comments below I managed to generate libpython27.a. However after following the advice on this post (2nd to last) I still had a the __imp_Py_InitModule4_64 error. After some serious Google-fu I managed to trip over this post telling me to rename the Py_InitModule4 line to Py_InitModule4_64. After that everything worked swimmingly.

Caribou

You need to create libpython27.a with gendef.exe python27.dll and dlltool.exe --dllname python27.dll --def python27.def --output-lib libpython27.a and place it in C:\Python27\libs – Wrath

I saw some posts mentioning gendef. But it's not easy finding information on how to go about generation the libpython27.a file. Would you happen to know of any? – Caribou

Oh ok, gendef and dlltool are two different commands. Will try. Thanks. – Caribou

Ok it compiled all right, but when I try to import the module i get Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: DLL load failed: The specified procedure could not be found. I think somewhere I must be missing a file. The only thing I did with the gendef stuff was to put libpython27.a in C:\Python27\libs. Did I miss something? – Caribou

So i went through all the steps as per this post (2nd from bottom). But now it won't compile, and i get this build\temp.win-amd64-2.7\Release\cquant.o:cQuant.c:(.text+0x1f09): undefined reference to __imp_Py_InitModule4_64' collect2.exe: error: ld returned 1 exit status` – Caribou

Hmm so is there anything that can be done about it, or will I have to admit defeat? – Caribou

I tried the -DMS_WIN64 flag too, but I still get the error. I don't really understand why it should be so hard to compile to 64 bit Python when 32 bit is very simple and straightforward. Thanks for the help though, much appreciated. – Caribou

I finally managed to sort it out. I just had to rename the Py_InitModule4 line to Py_InitModule4_64 in the python27.def file. Thanks a lot for the help. If you happen to be the cgohlke behind this site, thanks for that one too, those binaries have been a huge help from time to time :) – Caribou

Why are you hard set on mingw-w64? can you use visual c++ 2008 express sp1 with windows sdk 7 for .net frameworks 3.5 sp1 and the use setenv /release /x64, set DISTUTILS_USE_SDK=1, set MSSdk=1 and pip install blah or the install method of your choice. – Factfinding

I don't remember the exact details, but there was something with my code that Visual C++ 2008 didn't like (C99 related I think). Ordinarily I would just change the code, but the same code is used on a Linux box as well. So long story short I set the mingw-w64 environment up instead so that I could have the same code base throughout. – Caribou

Answer 1

This worked for me with Python 3.3 :

1. create static python lib from dll

   python dll is usually in C:/Windows/System32; in msys shell:

   gendef.exe python33.dll
   
   dlltool.exe --dllname python33.dll --def python33.def --output-lib libpython33.a
   
   mv libpython33.a C:/Python33/libs
   

2. use swig to generate wrappers

   e.g., swig -c++ -python myExtension.i

3. wrapper MUST be compiled with MS_WIN64, or your computer will crash when you import the class in Python

   g++ -c myExtension.cpp -I/other/includes
   
   g++ -DMS_WIN64 -c myExtension_wrap.cxx -IC:/Python33/include
   

4. shared library

   g++ -shared -o _myExtension.pyd myExtension.o myExtension_wrap.o -lPython33 -lOtherSharedLibs -LC:/Python33/libs -LC:/path/to/other/shared/libs
   

5. make sure all shared libs (gdal, OtherSharedLibs) are in your PATH (windows does not use LD_LIBRARY_PATH or PYTHONPATH)

6. in Python, just: import myExtension

voila!

Ant

Answer 2

I realize this is an old question, but it is still the top search result. Today, in 2019, I was able to do this:

https://github.com/PetterS/quickjs/commit/67bc2428b8c0716538b4583f4f2b0a2a5a49106c

In short:

1. Make sure a 64-bit version of mingw-w64 is in the PATH.
2. Monkey-patch distutils:

   import distutils.cygwinccompiler
   distutils.cygwinccompiler.get_msvcr = lambda: []
   

3. Some differences in the shell w.r.t. escaping.

4. extra_link_args = ["-Wl,-Bstatic", "-lpthread"] in order to link statically and not have extra runtime deps.

5. pipenv run python setup.py build -c mingw32 now works.

Conservationist

Thanks, yes it has been working nicely for me for a while now. We should probably try to get something more permanent checked into distutils.... – Conservationist

Answer 3

Here is a example code for VC++ Build Tools https://github.com/starnight/python-c-extension/tree/master/00-HelloWorld

You could try:

python setup.py -c mingw32

However this is not work for me.

My Solution is:

1. install Anaconda 64bit python 3.6

2. install mingw64

3. add mingw64/bin to PATH

4. compile dll from c file by

   gcc -c libmypy.c -IC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\include  
   gcc -shared -o libmypy.dll libmypy.o  -LC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\libs -lPython36
   

5. load dll file in .py script

   from ctypes import *  
   m = cdll.LoadLibrary(r"C:\{path_to_dll}\libmypy.dll")  
   print(m.hello())
   

Lifeless

Answer 4

I created a monkey-patch for setuptools to let you to build_ext with mingw64 on Windows easily. See https://github.com/imba-tjd/mingw64ccompiler

Bronchus

Answer 5

I used this thread to wade through learning how to make a C extension, and since most of what I learned is in it, I thought I'd put the final discovery here too, so that someone else can find it if they are looking.

I wasn't trying to compile something big, just the example in Hetland's Beginning Python. Here is what I did (the example C pgm is called palindrome.c). I'm using Anaconda with python 3.7 in it, and the TDM-GCC version of MinGW64. I put all of the tools used into my Path, and all of the paths needed in PYTHONPATH, and the ..\Anaconda3 directory into PYTHON_HOME. I still ended up using explicit paths on some things.

I created the libpython37.a library with gendef.exe and dlltool.exe as Mark said above, and put it in ..\Anaconda3\libs.

I followed the prescription in Hetland:

gcc -c palindrome.c

gcc -I$PYTHON_HOME -I$PYTHON_HOME/Include -c palindrome_wrap.c

The second failed, the compiler couldn't find Python.h, the following worked:

gcc -I[somedirectories]\Anaconda3\Include -c palindrome_wrap.c

I then did, as many have said, including Hetland 3rd ed.,

gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.dll

This did not work. Even with the Load Library cswu used (which I found elsewhere, too).

So I gendef'd _palindrome.dll and couldn't find the function in it, "is_palindrome" in the exports. I went through some of the SWIG documentation, and declared the function both in the %{ %} section and below it, both extern, that finally got the function extern'd in palindrome_wrap.c as it should have been. But no export, so I went back into palindrome.c and redeclared the function as:

declspec(dllexport) extern int __stdcall is_palindrome(char* text)

and redeclared it in palindrome.i in both places as above with this signature.

Partial success! It got listed in the Export section when I gendef'd _palindrome.dll and I could do cswu's call using Load Library. But still not do what Hetland says and do

import _palindrome

in Python.

Going back to all the sources again, I could not figure this out. I finally started reading the SWIG documentation from the beginning leaving no stone unturned -- Searching through the manual doesn't produce the place found.

At the end of Introduction Sec. 2.7 Incorporating Into a Build System, under the sample Make process, it says:

"The above example will generate native build files such as makefiles, nmake files and Visual Studio projects which will invoke SWIG and compile the generated C++ files into _example.so (UNIX) or _example.pyd (Windows). For other target languages on Windows a dll, instead of a .pyd file, is usually generated."

And that's the answer to the last problem:

The compile step for the dll should read:

gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.pyd

(I didn't go back and change out my declspec declarations so I don't know whether they were necessary, so they were still there too).

I got a file, _palindrome.pyd

Which if in the PYTHONPATH (mine was local) works, and one can then do

import _palindrome

from _palindrome import is_palindrome

and use the exported, properly wrapped and packaged C function, compiled with TDM-GCC, in python as promised. gcc, which is MinGW64 in a different installation, knows how to do the .pyd file. I diffed the dll and pyd since they were the same byte length. They are not the same at hundreds of points.

Hope this helps someone else.

Allogamy