Calling C/C++ from Python? [closed]

Question

What would be the quickest way to construct a Python binding to a C or C++ library?

(I am using Windows if this matters.)

Hanhhank

Answer 1

You should have a look at Boost.Python. Here is the short introduction taken from their website:

The Boost Python Library is a framework for interfacing Python and C++. It allows you to quickly and seamlessly expose C++ classes functions and objects to Python, and vice-versa, using no special tools -- just your C++ compiler. It is designed to wrap C++ interfaces non-intrusively, so that you should not have to change the C++ code at all in order to wrap it, making Boost.Python ideal for exposing 3rd-party libraries to Python. The library's use of advanced metaprogramming techniques simplifies its syntax for users, so that wrapping code takes on the look of a kind of declarative interface definition language (IDL).

Mahican

Boost.Python is one of the more user-friendly libraries in Boost, for a simple function call API it is quite straightforward and provides boilerplate you'd have to write yourself. It's a bit more complicated if you want to expose an object-oriented API. – Goodman

Boost.Python is the worst thing imaginable. For every new machine and with every upgrade it goes with linking problems. – Baldridge

Nearly 11 years later time for some contemplation about the quality of this answer? – Folkways

Is this still the best approach to interface python and c++ ? – Vaduz

Maybe you can try pybind11 which is lightweight compared to boost. – Assay

Answer 2

ctypes module is part of the standard library, and therefore is more stable and widely available than swig, which always tended to give me problems.

With ctypes, you need to satisfy any compile time dependency on python, and your binding will work on any python that has ctypes, not just the one it was compiled against.

Suppose you have a simple C++ example class you want to talk to in a file called foo.cpp:

#include <iostream>

class Foo{
    public:
        void bar(){
            std::cout << "Hello" << std::endl;
        }
};

Since ctypes can only talk to C functions, you need to provide those declaring them as extern "C"

extern "C" {
    Foo* Foo_new(){ return new Foo(); }
    void Foo_bar(Foo* foo){ foo->bar(); }
}

Next you have to compile this to a shared library

g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so  foo.o

And finally you have to write your python wrapper (e.g. in fooWrapper.py)

from ctypes import cdll
lib = cdll.LoadLibrary('./libfoo.so')

class Foo(object):
    def __init__(self):
        self.obj = lib.Foo_new()

    def bar(self):
        lib.Foo_bar(self.obj)

Once you have that you can call it like

f = Foo()
f.bar() #and you will see "Hello" on the screen

Amasa

This is pretty much what boost.python does for you in a single function call. – Crux

ctypes is in the python standard library, swig and boost are not. Swig and boost rely on extension modules and are therefore tied to python minor versions which indepentent shared objects are not. building a swig or boost wrappers can be a pain, ctypes makes no build requirements. – Sb

boost relies on voodoo template magic and an entirely custom build system, ctypes relies on simplicity. ctypes is dynamic, boost is static. ctypes can handle different versions of libraries. boost cannot. – Sb

mgb: but you get me riled over boost, I'd encourage you to post an answer yourself. However to show boosts superiority over ctypes, it'll have to be the same example, less then 4 lines C++ wrapper code, less then 2 lines of build instructions and no lines of python, oh and fit it on one screen too. – Sb

Agreed: I too have troubles with swig and boost. You should make clear in your answer which ones are the files and which name one should get them. – Refutation

On Windows I've had to specify __declspec(dllexport) in my function signatures for Python to be able to see them. From the above example this would correspond to: extern "C" { __declspec(dllexport) Foo* Foo_new(){ return new Foo(); } __declspec(dllexport) void Foo_bar(Foo* foo){ foo->bar(); } } – Fisk

What about passing arguments to bar()? Is this possible to do? For example, suppose you wanted to pass a python dict into bar() and have bar() iterate through and print each item. – Sporogony

Have a look at the ctypes reference linked in the answer. Basically you have to manually convert all arguments into the basic types known to C, else they are interpreted as int. You can define your own structs manually. Variable types as in a python dict are not supported, but of course you can use hack with void pointers if absolutely needed. Example for a double argument: lib.Foo_baz(self.obj, ctypes.c_double(3.5)) – Isolated

+1 Awesome, thanks! Maybe this is common knowledge but, just for posterity, I was using osx and had to replace -soname with -install_name to compile. – Persons

This example needs restype and argtypes set on the function pointers. The default restype is c_int, which truncates a 64-bit pointer. With restype=c_void_p, the pointer value is protected but still converted to a Python integer. As an argument this defaults to c_int again, unless argtypes is specified or it's manually re-wrapped in c_void_p. – Rockabilly

@DustinBoswell: I extended the code of this answer to include functions with double and integer parameters and integer return value - code is on pastebin (pastebin.com/pyUXf3RW) and (pastebin.com/0D700WPb). I don't know however where to put the function.restype=c_double to return a double, to not say how to pass a STL vector < vector <double> > parameter that is my ultimate need.. – Mensa

The link to ctypes in this answer is broken: docs.python.org/lib/module-ctypes.html – Luxury

Don't forget to delete the pointer afterwards by e.g. providing a Foo_delete function and calling it either from a python destructor or wrapping the object in a resource. – Joktan

I'm getting OSError: ./libfoo.so: undefined symbol: XGetImage, any idea what might be wrong? – Ergosterol

For dylib compilation on Mac, you would want: g++ -dynamiclib -undefined suppress -flat_namespace *.o -o something.dylib, as: #3533089 – Blameful

i got error that -soname not found so i ran this command instead. "g++ -fPIC -shared -o libfoo.so foo.o" – Pyriphlegethon

a similar example/tutorial with a bit more functionality: auctoris.co.uk/2017/04/29/… – Klina

I ended up getting a segfault inside my version of foo->bar() when accessing a member variable of the class (not in the above example). In this case adding argtypes and restype helped. See here #13754720 – Aikoail

As of February 2022, this answer is still the best option. Here's an answer that shows how to pass variables from Python to C. – Premium

I have done everything here, but I still get the error... FileNotFoundError: Could not find module 'path/to/libfoo.so' (or one of its dependencies). Try using the full path with constructor syntax. – Yolondayon

Answer 3

There is also pybind11, which is like a lightweight version of Boost.Python and compatible with all modern C++ compilers:

https://pybind11.readthedocs.io/en/latest/

Anthropogenesis

Today!! 2020 This should be the top answer! It is a template header only library. Many huge relevant projects recommends it, like Pytorch pytorch.org/tutorials/advanced/cpp_extension.html Also fully works on VS Community Windows – Slaw

Please also include some examples, to make it more useful than a link-only answer. – Somato

I've shown a minimal runnable example at: https://mcmap.net/q/64172/-calling-c-c-from-python-closed to showcase how awesome it is. – Broucek

Answer 4

The quickest way to do this is using SWIG.

Example from SWIG tutorial:

/* File : example.c */
int fact(int n) {
    if (n <= 1) return 1;
    else return n*fact(n-1);
}

Interface file:

/* example.i */
%module example
%{
/* Put header files here or function declarations like below */
extern int fact(int n);
%}

extern int fact(int n);

Building a Python module on Unix:

swig -python example.i
gcc -fPIC -c example.c example_wrap.c -I/usr/local/include/python2.7
gcc -shared example.o example_wrap.o -o _example.so

Usage:

>>> import example
>>> example.fact(5)
120

Note that you have to have python-dev. Also in some systems python header files will be in /usr/include/python2.7 based on the way you have installed it.

From the tutorial:

SWIG is a fairly complete C++ compiler with support for nearly every language feature. This includes preprocessing, pointers, classes, inheritance, and even C++ templates. SWIG can also be used to package structures and classes into proxy classes in the target language — exposing the underlying functionality in a very natural manner.

Pentathlon

For whoever having the problem <Python.h> not found , you need to do gcc $(pkg-config --cflags python3) -fPIC -c example.c example_wrap.c -I/usr/local/include/python3.10 . – Lasala

Answer 5

I started my journey in the Python <-> C++ binding from this page, with the objective of linking high level data types (multidimensional STL vectors with Python lists) :-)

Having tried the solutions based on both ctypes and boost.python (and not being a software engineer) I have found them complex when high level datatypes binding is required, while I have found SWIG much more simple for such cases.

This example uses therefore SWIG, and it has been tested in Linux (but SWIG is available and is widely used in Windows too).

The objective is to make a C++ function available to Python that takes a matrix in form of a 2D STL vector and returns an average of each row (as a 1D STL vector).

The code in C++ ("code.cpp") is as follow:

#include <vector>
#include "code.h"

using namespace std;

vector<double> average (vector< vector<double> > i_matrix) {

  // Compute average of each row..
  vector <double> averages;
  for (int r = 0; r < i_matrix.size(); r++){
    double rsum = 0.0;
    double ncols= i_matrix[r].size();
    for (int c = 0; c< i_matrix[r].size(); c++){
      rsum += i_matrix[r][c];
    }
    averages.push_back(rsum/ncols);
  }
  return averages;
}

The equivalent header ("code.h") is:

#ifndef _code
#define _code

#include <vector>

std::vector<double> average (std::vector< std::vector<double> > i_matrix);

#endif

We first compile the C++ code to create an object file:

g++ -c -fPIC code.cpp

We then define a SWIG interface definition file ("code.i") for our C++ functions.

%module code
%{
#include "code.h"
%}
%include "std_vector.i"
namespace std {

  /* On a side note, the names VecDouble and VecVecdouble can be changed, but the order of first the inner vector matters! */
  %template(VecDouble) vector<double>;
  %template(VecVecdouble) vector< vector<double> >;
}

%include "code.h"

Using SWIG, we generate a C++ interface source code from the SWIG interface definition file..

swig -c++ -python code.i

We finally compile the generated C++ interface source file and link everything together to generate a shared library that is directly importable by Python (the "_" matters):

g++ -c -fPIC code_wrap.cxx  -I/usr/include/python2.7 -I/usr/lib/python2.7
g++ -shared -Wl,-soname,_code.so -o _code.so code.o code_wrap.o

We can now use the function in Python scripts:

#!/usr/bin/env python

import code
a= [[3,5,7],[8,10,12]]
print a
b = code.average(a)
print "Assignment done"
print a
print b

Mensa

A real case implementation where in the C++ code stl vectors are passed as non const references and hence available by python as output parameters: lobianco.org/antonello/personal:portfolio:portopt – Mensa

Answer 6

For modern C++, use cppyy: http://cppyy.readthedocs.io/en/latest/

It's based on Cling, the C++ interpreter for Clang/LLVM. Bindings are at run-time and no additional intermediate language is necessary. Thanks to Clang, it supports C++17.

Install it using pip:

    $ pip install cppyy

For small projects, simply load the relevant library and the headers that you are interested in. E.g. take the code from the ctypes example is this thread, but split in header and code sections:

    $ cat foo.h
    class Foo {
    public:
        void bar();
    };

    $ cat foo.cpp
    #include "foo.h"
    #include <iostream>

    void Foo::bar() { std::cout << "Hello" << std::endl; }

Compile it:

    $ g++ -c -fPIC foo.cpp -o foo.o
    $ g++ -shared -Wl,-soname,libfoo.so -o libfoo.so  foo.o

and use it:

    $ python
    >>> import cppyy
    >>> cppyy.include("foo.h")
    >>> cppyy.load_library("foo")
    >>> from cppyy.gbl import Foo
    >>> f = Foo()
    >>> f.bar()
    Hello
    >>>

Large projects are supported with auto-loading of prepared reflection information and the cmake fragments to create them, so that users of installed packages can simply run:

    $ python
    >>> import cppyy
    >>> f = cppyy.gbl.Foo()
    >>> f.bar()
    Hello
    >>>

Thanks to LLVM, advanced features are possible, such as automatic template instantiation. To continue the example:

    >>> v = cppyy.gbl.std.vector[cppyy.gbl.Foo]()
    >>> v.push_back(f)
    >>> len(v)
    1
    >>> v[0].bar()
    Hello
    >>>

Note: I'm the author of cppyy.

Intraatomic

It doesn't, really: Cython is a Python-like programming language to write C extension modules for Python (the Cython code gets translated into C, together with the necessary C-API boilerplate). It provides some basic C++ support. Programming with cppyy only involves Python and C++, no language extensions. It's fully run-time and does not generate offline code (lazy generation scales a lot better). It targets modern C++ (incl. automatic template instantiations, moves, initializer_lists, lambda's, etc., etc.) and PyPy is supported natively (i.e. not through the slow C-API emulation layer). – Intraatomic

This PyHPC'16 paper contains a range of benchmark numbers. Since then, there have been definite improvements on the CPython side, though. – Intraatomic

I like this approach because you don't have to do additional integration work with swig, ctypes, or boost.python. Instead of you having to write code to get python to work with your c++ code... python does the hard work to figure out c++. Assuming it actually works. – Cranberry

cppyy is very interesting! I see in the docs that redistribution and pre-packaging is handled. Is this known to work well with tools that package python code as well (e.g., PyInstaller)? And is this related to the ROOT project, or leverage its work? – Cymoid

Thanks! I'm not familiar with PyInstaller, but the "dictionaries" that package up forward declarations, paths and headers are C++ codes compiled to shared libs. Since cppyy is used to bind C++ code, I presume handling some more C++ code should be fine. And that code is not dependent on the Python C-API (only the libcppyy module is), which simplifies things. cppyy itself can be installed from conda-forge or pypi (pip), so any of those environments work, for sure. Yes, cppyy started as a fork from PyROOT, but it has since improved so much, that the ROOT team is rebasing PyROOT on top of cppyy. – Intraatomic

Answer 7

pybind11 minimal runnable example

pybind11 was previously mentioned at https://mcmap.net/q/64172/-calling-c-c-from-python-closed but I would like to give here a concrete usage example and some further discussion about implementation.

All and all, I highly recommend pybind11 because it is really easy to use: you just include a header and then pybind11 uses template magic to inspect the C++ class you want to expose to Python and does that transparently.

The downside of this template magic is that it slows down compilation immediately adding a few seconds to any file that uses pybind11, see for example the investigation done on this issue. PyTorch agrees. A proposal to remediate this problem has been made at: https://github.com/pybind/pybind11/pull/2445

Here is a minimal runnable example to give you a feel of how awesome pybind11 is:

class_test.cpp

#include <string>

#include <pybind11/pybind11.h>

struct ClassTest {
    ClassTest(const std::string &name, int i) : name(name), i(i) { }
    void setName(const std::string &name_) { name = name_; }
    const std::string getName() const { return name + "z"; }
    void setI(const int i) { this->i = i; }
    const int getI() const { return i + 1; }
    std::string name;
    int i;
};

namespace py = pybind11;

PYBIND11_PLUGIN(class_test) {
    py::module m("my_module", "pybind11 example plugin");
    py::class_<ClassTest>(m, "ClassTest")
        .def(py::init<const std::string &, int>())
        .def("setName", &ClassTest::setName)
        .def("getName", &ClassTest::getName)
        .def_readwrite("name", &ClassTest::name)
        .def("setI", &ClassTest::setI)
        .def("getI", &ClassTest::getI)
        .def_readwrite("i", &ClassTest::i);
    return m.ptr();
}

class_test_main.py

#!/usr/bin/env python3

import class_test

my_class_test = class_test.ClassTest("abc", 1);
print(my_class_test.getName())
print(my_class_test.getI())
my_class_test.setName("012")
my_class_test.setI(2)
print(my_class_test.getName())
print(my_class_test.getI())
assert(my_class_test.getName() == "012z")
assert(my_class_test.getI() == 3)

Compile and run:

#!/usr/bin/env bash
set -eux
sudo apt install pybind11-dev
g++ `python3-config --cflags` -shared -std=c++11 -fPIC class_test.cpp \
  -o class_test`python3-config --extension-suffix` `python3-config --libs`
./class_test_main.py

Stdout output:

abcz
2
012z
3

If we tried to use a wrong type as in:

my_class_test.setI("abc")

it blows up as expected:

Traceback (most recent call last):
  File "/home/ciro/test/./class_test_main.py", line 9, in <module>
    my_class_test.setI("abc")
TypeError: setI(): incompatible function arguments. The following argument types are supported:
    1. (self: my_module.ClassTest, arg0: int) -> None

Invoked with: <my_module.ClassTest object at 0x7f2980254fb0>, 'abc'

This example shows how pybind11 allows you to effortlessly expose the ClassTest C++ class to Python!

Notably, Pybind11 automatically understands from the C++ code that name is an std::string, and therefore should be mapped to a Python str object.

Compilation produces a file named class_test.cpython-36m-x86_64-linux-gnu.so which class_test_main.py automatically picks up as the definition point for the class_test natively defined module.

Perhaps the realization of how awesome this is only sinks in if you try to do the same thing by hand with the native Python API, see for example this example of doing that, which has about 10x more code: https://github.com/cirosantilli/python-cheat/blob/4f676f62e87810582ad53b2fb426b74eae52aad5/py_from_c/pure.c On that example you can see how the C code has to painfully and explicitly define the Python class bit by bit with all the information it contains (members, methods, further metadata...). See also:

• Can python-C++ extension get a C++ object and call its member function?
• Exposing a C++ class instance to a python embedded interpreter
• A full and minimal example for a class (not method) with Python C Extension?
• Embedding Python in C++ and calling methods from the C++ code with Boost.Python
• Inheritance in Python C++ extension

pybind11 claims to be similar to Boost.Python which was mentioned at https://mcmap.net/q/64172/-calling-c-c-from-python-closed but more minimal because it is freed from the bloat of being inside the Boost project:

pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent Boost.Python library by David Abrahams: to minimize boilerplate code in traditional extension modules by inferring type information using compile-time introspection.

The main issue with Boost.Python—and the reason for creating such a similar project—is Boost. Boost is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available, this heavy machinery has become an excessively large and unnecessary dependency.

Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn't relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depend on Python (2.7 or 3.x, or PyPy2.7 >= 5.7) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (specifically: tuples, lambda functions and variadic templates). Since its creation, this library has grown beyond Boost.Python in many ways, leading to dramatically simpler binding code in many common situations.

pybind11 is also the only non-native alternative hightlighted by the current Microsoft Python C binding documentation at: https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2019 (archive).

Tested on Ubuntu 18.04, pybind11 2.0.1, Python 3.6.8, GCC 7.4.0.

Broucek

Answer 8

I think cffi for python can be an option.

The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn.

http://cffi.readthedocs.org/en/release-0.7/

Superstratum

I think this can only call c (not c++), still +1 (I really like cffi). – Osmond

Answer 9

I love cppyy, it makes it very easy to extend Python with C++ code, dramatically increasing performance when needed.

It is powerful and frankly very simple to use,

here it is an example of how you can create a numpy array and pass it to a class member function in C++.

cppyy_test.py

import cppyy
import numpy as np
cppyy.include('Buffer.h')


s = cppyy.gbl.Buffer()
numpy_array = np.empty(32000, np.float64)
s.get_numpy_array(numpy_array.data, numpy_array.size)
print(numpy_array[:20])

Buffer.h

struct Buffer {
  void get_numpy_array(double *ad, int size) {
    for( long i=0; i < size; i++)
        ad[i]=i;
  }
};

You can also create a Python module very easily (with CMake), this way you will avoid recompile the C++ code all the times.

Osber

Answer 10

The question is how to call a C function from Python, if I understood correctly. Then the best bet are Ctypes (BTW portable across all variants of Python).

>>> from ctypes import *
>>> libc = cdll.msvcrt
>>> print libc.time(None)
1438069008
>>> printf = libc.printf
>>> printf("Hello, %s\n", "World!")
Hello, World!
14
>>> printf("%d bottles of beer\n", 42)
42 bottles of beer
19

For a detailed guide you may want to refer to my blog article.

Naturalize

It may be worth noting that while ctypes are portable, your code requires a Windows-specific C library. – Armistice

Answer 11

Cython is definitely the way to go, unless you anticipate writing Java wrappers, in which case SWIG may be preferable.

I recommend using the runcython command line utility, it makes the process of using Cython extremely easy. If you need to pass structured data to C++, take a look at Google's protobuf library, it's very convenient.

Here is a minimal examples I made that uses both tools:

https://github.com/nicodjimenez/python2cpp

Hope it can be a useful starting point.

Olives

Answer 12

First you should decide what is your particular purpose. The official Python documentation on extending and embedding the Python interpreter was mentioned above, I can add a good overview of binary extensions. The use cases can be divided into 3 categories:

• accelerator modules: to run faster than the equivalent pure Python code runs in CPython.
• wrapper modules: to expose existing C interfaces to Python code.
• low level system access: to access lower level features of the CPython runtime, the operating system, or the underlying hardware.

In order to give some broader perspective for other interested and since your initial question is a bit vague ("to a C or C++ library") I think this information might be interesting to you. On the link above you can read on disadvantages of using binary extensions and its alternatives.

Apart from the other answers suggested, if you want an accelerator module, you can try Numba. It works "by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool)".

Siddons