Compile main Python program using Cython

Question

I have a Python2.6 program that can load Python modules compiled to .so files using Cython. I used Cython to compile the .py modules to .so files and everything works fine.

This is the setup.py file I use with Cython:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension("ldap", ["ldap.pyx"]),
    Extension("checker", ["checker.pyx"]),
    Extension("finder", ["finder.pyx"]),
    Extension("utils", ["utils.pyx"]),
]

setup(
  name = 'bchecker',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)

So I know I can compile Python modules using Cython (I guess Cython creates 'C' files from my Python files and then compiles them), but can I compile my main Python program to something I can execute on a Linux platform? If so, a Cython command line example would be appreciated. Thanks.

Maddocks

Answer 1

Contrary to what Adam Matan and others assert, you can in fact create a single executable binary file using Cython, from a pure Python (.py) file.

Yes, Cython is intended to be used as stated - as a way of simplifying writing C/C++ extension modules for the CPython python runtime.

But, as nudzo alludes to in this comment, you can use the --embed switch at the command line prompt.

Here is an extremely simple example. I am peforming this from a Debian Sid workstation, using python3 and cython3..

Make sure you have python-dev or python3-dev packages installed beforehand.

1) Create a very simple Python program called hello.py

$ cat hello.py

print("Hello World!")

2) Use Cython to compile your python program into C...

cython3 --embed -o hello.c hello.py

3) Use GCC to compile hello.c into an executable file called hello...

gcc -Os -I /usr/include/python3.3m -o hello hello.c -lpython3.3m -lpthread -lm -lutil -ldl

4) You end up with a file called hello ...

$ file hello

hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=006f45195a26f1949c6ed051df9cbd4433e1ac23, not stripped

$ ldd hello
linux-vdso.so.1 (0x00007fff273fe000)
libpython3.3m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.3m.so.1.0 (0x00007fc61dc2c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc61da0f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc61d70b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fc61d508000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc61d304000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc61cf5a000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc61cd52000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fc61cb28000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc61c90f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc61e280000)

In this case, the executable is dynamically linked to Python 3.3 on my Debian system.

5) run hello...

$ ./hello

Hello World!

As you can see, using this method you can basically use Cython to convert your pure Python applications into executable, compiled object code.

I am using this method for vastly more complex applications - for example, a full blown Python/PySide/Qt application.

For different versions of Python, you tailor the gcc -I and -l switches to suit.

You can then package the executable as a distribution (.deb, etc.) file, without having to package the Python/PySide/Qt files - the advantage being that your application should still be able to run even after a distribution update to the same versions of Python, etc. on that distribution.

Daveta

It does not compiles module(s) into the executable. Is it possible to have a portable executable i.e every module which had been imported in source code would be compiled into that executable. @BrokenMan – Boren

@DeveshSaini I haven't spent the time trying to figure that bit out. The only way I've managed to get a full-blown python+Pyside application to compile as above, was to develop the program in the normal fashion - lots of include <mymodule> statements + testing with the interpreter. Then when all is ready I manually paste in all the included code and comment out the relevent include statements they replaced. Yes, it's undesirable, but like I said at first I've not spent the time to see how I can compile just the main routine and link the included modules together. – Daveta

@BrokenMan What is the meaning of this error when I try to compile with GCC in windows? Python header needed to compile C extensions, please install development version of Python – Eboni

@Eboni Looks like you haven't yet installed the python-dev package (or python3-dev). You need that/those. I've added in a note about that in my post above. – Daveta

@BrokenMan I am running Windows – Eboni

@Eboni I don't do Windows development (yet) - makes for a great gaming OS though ;) - but a quick google search that I've performed on your behalf comes up with this SO topic : #16449210 which may help you out. – Daveta

@Eboni Just realised I missed the bit in your original question mentioning "GCC in windows" - my apologies, not enough coffee it seems. Anyway, docs.python.org/2/using/windows.html might help you out, which links to python.org/downloads/windows - and also try googling for "windows python-dev" which might provide you with further helpful information. If you find a solution to your problem, it might be a good idea to post your solution here. Rgds. – Daveta

What about static linking of libpython? – Lapidate

@JamesMills This might help you : #8367044 – Daveta

@Broken Man i have Anconda python3, i used your step (3) "gcc -Os -I /usr/include/python3.3m -o hello hello.c -lpython3.3m -lpthread -lm -lutil -ldl" and used the Anconda include directory in the argument but can not include Lpython, how to do that? and could you explain step3 code in detail. – Fugacious

@Fugacious I have never used Anaconda python so don't know how it works. Perhaps it would make for a standalone question in its own right, referring to this answer? – Daveta

Can you elaorate on how to convert py to c++ file and then to a binary?..In terms of command lines? – Clementia

@BrokenMan I have a Cython post you may be able to provide insight on. – Auriculate

@HaydenDarcy the example given is for Debian Linux. I haven't tested it in Windows. – Daveta

For python 2.7 cython --embed -o /tmp/a.c a.py and gcc -Os -I /usr/include/python2.7 -o /tmp/a/tmp/a.c -lpython2.7 -lpthread -lm -lutil -ldl – Treen

Answer 2

Take a look at the answers to Can Cython compile to an EXE? which, contrary to all the other answers here say that yes, it is possible to compile to an executable.

The links at Embedding Cython seem to be a good place to start, but it's not Cython's primary purpose so I don't know how straightforward it would be.

Eddyede

Answer 3

I don't know if this will help or not but Nudzo is correct. You can get it with cython --embed -o main.o main.py and then I try to compile the result with cl/EHsc

Dorene

Answer 4

Look at this Post:

cython <cython_file> --embed

and then just

gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

here an example:

cython3 main.py --embed
gcc main.c -I /usr/include/python3.8/ -L /lib/x86_64-linux-gnu/ -l python3.8 -o main

Lal

Answer 5

You can't, Cython is not made to compile Python nor turn it into an executable.

To produce an .exe file, use py2exe.

To produce a package for Mac or Linux, use the regulard packaging system process as there is nothing specific about a script language program in Unix env.

Cesar

Well it allows me to turn my .py files into .so files. See the above script I use. Don't the .so files contain executable code? – Maddocks

It's not the purpose. Cython is for compiling C/C++ Python extensions, not turn Python scripts to executable. – Cesar

Misinformation. There's cython --embed which can produce executables with embed Python. – Meneau

The resulting executable still depends on libpython unless you link it statically. Plus it will not produce one executable out of all your files, but one executable out of your entry point. – Cesar

@e-satis Yes, the executable will be dynamically linked to the version of libpython it was compiled against at the time. What's the issue? If it's for Linux, and if you're planning on distributing your application, you just package it to whatever linux distribution format you're targeting - just like every other binary package. e.g. myapp_1_wheezy_amd64.deb would be your compiled python app linked against either python2 or python3 for that distro. For Windows just supply the required python. Again, I fail to see the problem here, or the objection. – Daveta

I would have to highlight his point here. You are embedding the whole python interpreter into the entry point of your executables that then is being called and processing you application code as python. That is not what you need when you compile to affect performance of your application. – Axis