How to set python function as callback for c++ using pybind11?

Question

typedef bool (*ftype_callback)(ClientInterface* client, const Member* member ,int member_num);

struct Member{
    char x[64];
    int y;
};

class ClientInterface {
public: 
    virtual int calc()=0;
    virtual bool join()=0;
    virtual bool set_callback(ftype_callback on_member_join)=0;
};

It is from SDK which I can call the client from dynamic library in c++ codes.

bool cb(ClientInterface* client, const Member* member ,int member_num) {
    // do something
}
cli->set_callback(cb);
cli->join();

I want to port it to python bindings use pybind11. How do I set_callback in python?

I have seen the doc and try:

PYBIND11_MODULE(xxx, m) {
    m.def("set_callback", [](xxx &self, py::function cb ){
        self.set_callback(cb);
    });
}

The code just failed to compile.

My question, how do I convert the py::function to ftype_callback or there is other way to make it?

Eudora

Where do you get the cb? Do you define it in your C++ code or in Python code? – Cool

@Cool the example codes is defined in C++. I want to define callback function in Python and set_callback in Python so the SDK will call the `callback function in Python code. – Eudora

Have you ever seen the doc? Is it helpful? – Cool

Description has been updated. – Eudora

Not sure what you want is directly possible. The list of builtin conversions does not include function pointers. It includes std::function, which is much more flexible. If you can change the C++ code, consider changing ftype_callback to using ftype_callback = std::function<bool(ClientInterface*,const Member*,int)>; and use ftype_callback& in the lambda argument, similarly to the func_arg example in the documentation you linked to. – Balbur

Answer 1

You need a little C++ to get things going. I'm going to use a simpler structure to make the answer more readable. In your binding code:

#include <pybind11/pybind11.h>

#include <functional>
#include <string>

namespace py = pybind11;

struct Foo
{
    int i;
    float f;
    std::string s;
};

struct Bar
{
    std::function<bool(const Foo &foo)> python_handler;
    std::function<bool(const Foo *foo)> cxx_handler;

    Bar()
    {
        cxx_handler = [this](const Foo *foo) { return python_handler(*foo); };
    }
};

PYBIND11_MODULE(example, m)
{
    py::class_<Foo>(m, "Foo")  //
        .def_readwrite("i", &Foo::i)
        .def_readwrite("f", &Foo::f)
        .def_readwrite("s", &Foo::i);

    py::class_<Bar>(m, "Bar")  //
        .def_readwrite("handler", &Bar::python_handler);
}

Here, Foo is the object that is passed to the callback, and Bar is the object that needs its callback function set. Since you use pointers, I have wrapped the python_handler function with cxx_handler that is meant to be used in C++, and converted the pointer to reference.

To be complete, I'll give a possible example of usage of the module here:

import module.example as impl

class Bar:
    def __init__(self):
        self.bar = impl.Bar()
        self.bar.handler = self.handler
        
    def handler(self, foo):
        print(foo)
        return True

I have used this structure successfully in one of my projects. I don't know how you want to proceed, but perhaps if you don't want to change your original structure you can write wrapper classes upon them that use the given structure.

Update:

I thought that you controlled the structure when I wrote the answer above (I'll keep it for anyone who needs it). If you have a single cli instance, you can do something like:

using Handler = std::function<bool(std::string, int, int)>;

Handler handler;

bool cb(ClientInterface *client, const Member *member, int member_num)
{
    return handler(std::string(member->x), member->y, member_num);
}

// We have created cli somehow

// cli->set_callback(cb);

// cli->join();

PYBIND11_MODULE(example, m)
{
    m.def("set_callback", [](Handler h) { handler = h; });
}

If you have multiple ClientInterface instances, you can map ClientInterface pointers to handlers and call the appropriate handler in the cb function based on given ClientInterface pointer.

Note: I haven't tested the above with a python script but it should work.

Another Update

If you want to handle multiple instances, the code can roughly look like this:

using Handler = std::function<bool(std::string, int, int)>;

std::map<ClientInterface *, handler> map;

bool cb(ClientInterface *client, const Member *member, int member_num)
{
    // Check if <client> instance exists in map
    return map[client](std::string(member->x), member->y, member_num);
}

PYBIND11_MODULE(example, m)
{
    m.def("set_callback", [](int clientid, Handler h) 
    {
        // Somehow map <clientid> to <client> pointer
        map[client] = h;
    });
}

Note that this isn't a runnable code and you need to complete it.

Guidotti

Thanks for the answer. My question is that the set_callback is defined in the SDK to use function pointer. I am not allowed to change it. I can not figure out how should I use your pattern to tackle it. – Eudora

Just updated my answer to reflect your comment. – Guidotti

May I ask how to map multiple instances to different handler? I implement the callback function use this pattern. However, it seems the set_callback will set the callback among all the threads in python. Multiprocessing is fine. – Eudora

I edited the answer to include mapping demonstration. To handle multiple threads, maybe you can open one client per thread or something like that. It all depends on what you're trying to achieve. Hopefully the update will solve your problem. – Guidotti

Answer 2

you can get data using python types in pybind11

make sure you have #include <pybind11/functional.h>

// c++
using PyCallback = std::function<void(pybind11::bytearray)>;

class Haha
{
public:
    void setCallback(PyCallback& pyfn) {
        m_pyfn = pyfn;
    }
    void onDataAvaiable(char* buf, int len) {
        m_pyfn(pybind11::bytearray(buf, len));
    }
private:
    PyCallback m_pyfn;
};

PYBIND11_MODULE(haha, m) {
    pybind11::class_<Haha>(m, "Haha")
        .def("setCallback", &Haha::setCallback);
}

// python
def fn(data):
    print(data)

hahaInstance = m.Haha()
hahaInstance .setCallback(fn)
while True:
    // block to make sure hahaInstance is always running, then callback will print data

Assign

I think this answer could be improved by using similar identifiers and style conventions as the asker used in their code. – Alyciaalyda