Generate python bindings, what methods/programs to use [closed]

Question

I'm looking at using python (CPython) in my program to both allow user scripting in my environment and to allow me to use pyside, the qt bindings for c++ to create the GUI for my application. These can be effectively separated with the idea that the GUI python code can later be compiled away for speed (if that would be possible).

I'm very new to python and i'm really looking for the most efficient way to generate solid bindings with the absolute minimum of additional code to maintain as the bindings are likely to change often as the project evolves. I need it so that python classes extend c++ classes that have virtuals.

I have already looked into PyBindGen and it chokes too often on things in my library for it to be practically useful.

Any help/advice/links/workflows you recommend in this regard would be very helpful.

Club

IMHO your Python and C++ parts are then prone to be very intertwined with each other anyway, if you integrate to that level. But I support the quest for nice bindings in general. +1. – Closing

Answer 1

There are only a few projects I know to have automatic binding generators for C++. The first one is SWIG. As some other answer has already said, it is a little bit old style, but it works. The second one is Boost.Python - by itself, it does not generate the bindings automatically, but you can use Boost.Pyste to do it for you. It requires GCC-XML to parse your original source code and write the Boost.Python bindings. New entries in this universe are cppyy and Shiboken. All options do support virtual methods in C++ that can be overloaded from Python.

That said, I must complement that, normally, when you are binding, one should avoid blindly bind everything you have in C++ into Python - if you do it like this you may not get a very pythonic feeling from the Python side. Instead, you design how you'd like your library to be used in Python, in the most pythonic possible way, and then you walk back and see how to patch it through to your C++ code using one of the possible binding libraries. For example, instead of dealing with std::vector's, you'd prefer your library calls handle Python lists or iterables. If your C++ library receives a std::map, you'd like that is handled with Python dictionaries. If it is an array, maybe a numpy.ndarray would be more convenient. And so on...

That said, you can still design your bindings so that your maintenance is minimized.

Here is a list of other Python/C++ wrapping, in case you decide to look around a bit further:

1. SWIG - as you already know
2. Boost.Python - this is what we normally use around here - pretty well structured. pybind11, below, may be a more modern alternative.
3. Cython - very neat syntax close to Python - they claim to be much faster than Boost.Python
4. SIP - not very much spread, but it is there
5. pybind11 - Syntax similar to Boost.Python, compact implementation thanks to C++11.
6. cppyy - cppyy provides fully automatic, dynamic Python-C++ bindings by leveraging the Cling C++ interpreter and LLVM. It supports both PyPy (natively), CPython, and C++ language standards through C++17 (and parts of C++20).
7. Shiboken - is part of the Qt framework, and can extract information from C or C++ headers and generate CPython code that allow to bring C or C++ projects to Python.

These are now inactive:

1. PyBindGen - claims to be the fastest, but inactive since may 21st, 2017, previous release on 2014 - currently maintained on github (https://github.com/gjcarneiro/pybindgen/releases)
2. ECS:Python - inactive since december 6th, 2014 (v2.8), moved to github (https://github.com/MarcusTomlinson/ECS-Python)
3. PyCXX - C++ facilities to make it easier to write Python extensions - inactive? Last release was v7.0.2 on 23rd april 2017
4. CLIF - CLIF provides a common foundation for creating C++ wrapper generators for various languages - inactive? Not much activity on repo (only 14 commits)

For completeness, it is also possible to load compiled C code directly into Python without creating formal bindings. You can do this using FFI with any of these two Python modules:

1. ctypes - This native to Python and requires no external module installation.
2. cffi - This is a new package inspired by the equivalent design on the Lua JIT.

Reflation

A new addition to this list is pybind11 (github.com/wjakob/pybind11, pybind11.readthedocs.org/en/latest). Disclaimer: I'm the author. – Khanna

@WenzelJakob I think you should add a new answer with that information, and include the disclaimer in the body of that post. – Hotbed

Andre, if you'd like to consider adding the information suggested by Wenzel Jakob, do so (Revision 4 of this answer). – Hotbed

Please do it, apparently I don't have enough reputation... – Amontillado

There you are... I wasn't sure how to do the change, so the confusion. It is now in place. Thanks for passing by! – Amontillado

PyBindGen seems to be active again – Agglutinogen

What about Shiboken? – Clifford

Answer 2

If you're looking for speed, I would definitely vote for Cython too. As far as I know about other ways to interface C++ with Python, Cython is not that much heavy to maintain/update compared to other binding tools, regarding to the "fluidity of workflow" gain. (Cythonized code argues to be as fast as pure C, or not far from that, which also make it very interesting).

Anyway, there's several good API's for connecting python code with C++ (Boost.Python, ...), but I think that all of them will result in the need to expose methods directly in your C++ source code (You guys tell me if I'm wrong, or imprecise).

In the other hand, Cython will give you the possibility to keep your C++ API (GUI, or whatever...) strictly separated from the exposure sources (so called .pyx extensions). The final workflow would be :

C++ API => Compilation as shared object => Cython extension(s) (importing and exposing C++ features) => Compilation of the extension(s) => Using the extension (extension to be added to your python path).

The good news is that you'll have to maintain only the changing parts of your pool of .pyx files related to the evolving C++ features (among those which require to be exposed). It is kind of an investment at first, but, to my experience, once this workflow is setup, it is quite straight forward to make the whole stuff grow up in complexity.

Now concerning your need to extend classes that have virtuals and override them from Python (If I get correctly what you meant). It is doable. Once again, not so direct, but you should have a look at this thread.

The bad news : in that particular case you'll have to create some extra C++ adapters/interfaces, to enable the call of the parent method if the extended python one doesn't override the given parent's method. (Note that redefining a C++ exposed method, virtual or not, from python is a replacement of the function, but is absolutely not equivalent to an override).

Hummf, now that I am reading myself back, it looks a bit confusing. Hope this is still helpful.

I can be more specific about the workflow you'll have to deal with if you choose the Cython option, if you ask me to, but I think that the thread linked above is quite a good starting point...

Dreda

If you need any more details on wrapping C/C++ library with Cython, you may also find useful a post I wrote about it - it has all the details needed to wrap and distribute a C/C++ library: martinsosic.com/development/2016/02/08/… – Rachellerachis

Answer 3

For modern C++ use CLIF (https://github.com/google/clif). It does not discover your API, you need to describe it in Python terms.

CLIF will recompile your header with the latest LLVM/Clang compiler and produce C++ source code for a Python extension module.

It can understand complex data structures: def f() -> dict<str, set<int>>.

Smirk

Answer 4

For modern C++, use cppyy: http://cppyy.readthedocs.io/en/latest/

It's based on Cling, the C++ interpreter for Clang/LLVM. Bindings are at run-time and no additional intermediate language is necessary. Thanks to Clang, it supports C++17.

Specific to the original question of using Qt: several recent updates were done explicitly to support bindings generation for all of KDE.

For speed: cppyy is natively supported by PyPy.

Note: I'm the main author of cppyy.

Fidget

can cppyy easily be used with Qt based project? – Puncture

Yes, using the cmake fragments that were designed for KDE: cppyy.readthedocs.io/en/latest/… – Fidget

Answer 5

AFAICT, there aren't that many options with C++. One of the older projects is SWIG; it's reported to be a little arcane and clunky, but since it's been around a long time it should also cover a lot of stuff not handled by other projects. You might also want to look at Cython.

Finish

Answer 6

SWIG is a great tools that can do what you want

http://sourceforge.net/projects/swig/

Tonatonal