How to get the OpenCV image from Python and use it in C++ in pybind11?

Question

I'm trying to figure out how it is possible to receive an OpenCV image from a Python in C++. I'm trying to send a callback function, from C++ to my Python module, and then when I call a specific python method in my C++ app, I can access the needed image.

Before I add more details, I need to add that there are already several questions in this regard including :

1. how-to-convert-opencv-image-data-from-python-to-c
2. pass-image-data-from-python-to-cvmat-in-c
3. writing-python-bindings-for-c-code-that-use-opencv
4. c-conversion-from-numpy-array-to-mat-opencv

but none of them have anything about Pybind11. In fact they are all using the PyObject (from Python.h header) with and without Boost.Python. So my first attempt is to know how it is possible in Pybind11 knowing that it has support for Numpy arrays, so it can hopefully make things much easier.

Also On the C++ side, OpenCV has two versions, 3.x and 4.x which 4.x as I've recently found, is C++11 compliant. on Python side, I used OpenCV 3.x and I'm on a crossroad of which one to choose and what implications it has when it comes to Pybind11.

What I have tried so far: I made a quick dummy callback and tried passing a simple cv::Mat& like this :

#include <pybind11/embed.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>
namespace py = pybind11;
...

void cpp_callback1(bool i, std::string id, cv::Mat img)
{ 
    auto timenow = chrono::system_clock::to_time_t(chrono::system_clock::now());
    cout  <<"arg1: " << i << " arg2: " << id<<" arg3: " << typeid(img).name() <<" " << ctime(&timenow)<<endl;
}

and used it like this :

py::list callback_lst;
callback_lst.attr("append")(py::cpp_function(cpp_callback1));

py::dict core_kwargs = py::dict("callback_list"_a = callback_lst,
                                "debug_show_feed"_a = true);

py::object core_obj = core_cls(**core_kwargs);
core_obj.attr("start")();

but it fails with an exception on python part which says :

29/03/2020 21:56:47 : exception occured ("(): incompatible function arguments. The following argument types are supported:\n    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None\n\nInvoked with: True, '5', array([[[195, 217, 237],\n        [195, 217, 237],\n        [196, 218, 238],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       ...,\n\n       [[120, 129, 140],\n        [110, 120, 130],\n        [113, 122, 133],\n        ...,\n        [196, 209, 245],\n        [195, 207, 244],\n        [195, 207, 244]],\n\n       [[120, 133, 142],\n        [109, 121, 130],\n        [114, 120, 131],\n        ...,\n        [195, 208, 242],\n        [195, 208, 242],\n        [195, 208, 242]],\n\n       [[121, 134, 143],\n        [106, 119, 128],\n        [109, 114, 126],\n        ...,\n        [194, 207, 241],\n        [195, 208, 242],\n        [195, 208, 242]]], dtype=uint8)",) 
Traceback (most recent call last):
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 257, in start
    self._main_loop()
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 301, in _main_loop
    self._execute_callbacks(is_valid, name, frame)
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 142, in _execute_callbacks
    callback(*args)
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None

Invoked with: True, '5', array([[[195, 217, 237],
        [195, 217, 237],
        [196, 218, 238],
        ...,
        [211, 241, 255],
        [211, 241, 255],
        [211, 241, 255]],

       [[195, 217, 237],
        [195, 217, 237],
        [195, 217, 237],
        ...,

Using py::object or py::array_t<uint8_t> instead of cv::Mat doesn't cause any errors, but I can't seem to find a way to cast them back to a cv::Mat properly!

I tried to cast the numpy array into a cv::Mat as instructed in the comments but the output is garbage:

void cpp_callback1(bool i, std::string id, py::array_t<uint8_t>& img)
{ 
    auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    //py::buffer_info buf = img.request();
    cv::Mat img2(rows, cols, type, img.ptr());
    cv::imshow("test", img2);
}

results in :

It seems to me, the strides, or something in that direction is messed up that image is showing like this. what am I doing wrong here? I couldn't use the img.strides() though! when printed it using py::print, it shows 960 or something like that. So I'm completely clueless how to interpret that!

Turk

Have the callback take a numpy array instead of Mat. Then in the callback, create a Mat header for the numpy array's data buffer. (keep in mind that this shares the bufffer, so if you need the Mat to have longer lifetime than the callback's scope, you'll have to make a deep copy) – Malleus

@DanMašek , thanks a lot, but that do you mean py::array_t<double> ? – Turk

Yeah, looks like it. Although you probably have to change the template parameter to match the actual datatype of the numpy array -- if it's an image, it's probably uint8_t instead. – Malleus

@DanMašek Thanks a lot. I can really appreciate a snippet of some kind. I'm kind of lost here! specially concerning the Mat header! how should I do this? – Turk

This constructor (or one of the related ones). You have to extract the width and height and also channel count from the numpy array's shape, and get a pointer to its data buffer. -- looks like the direct access functionality in pybind let's you do that. – Malleus

I'd love to give you a complete code example, but I'd first have to set up PyBind and grok some of the details. Wouldn't mind getting bit more familiar with it, so that's not an issue, but at this particular moment, I've still got some work I need to do tonight... maybe later tonight. – Malleus

@DanMašek , Thanks a lot man. its really greatly appreciated. I'll get to it and hopefully get it solved. in case I faced something I'll update the question. – Turk

@DanMašek : I did this, but the resulting image is corrupted. could you please have a look and see if you can spot where I went wrong? thanks a lot in advance – Turk

The discussion here might be very useful and offers a way of doing this without hardcoding the type of the array – Cultch

Answer 1

I ultimately could successfully get this to work thanks to @DanMasek and this link:

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    py::buffer_info buf = img.request();
    cv::Mat mat(buf.shape[0], buf.shape[1], CV_8UC3, (unsigned char*)buf.ptr);
    
    cv::imshow("test", mat);
}

note that the cast is necessary, or otherwise, you'd get a blackish screen only!
However, if somehow there was a way like py::return_value_policy that we could use to change the type of reference, so even though the python part ends, the c++ side wouldn't crash would be great.

side note :
it seems the ptr property exposed in the numpy array, is actually not a py::handle but a PyObject*&. I couldn't have a successful conversion and thus resorted to the solution I posted above. I'll update this answer, when I figure this out.

Update:

I found out, the arrays data holds a pointer to the underlying buffer and can be used easily as well. From <pybind11/numpy.h> L681:

/// Pointer to the contained data. If index is not provided, points to the
/// beginning of the buffer. May throw if the index would lead to out of bounds access.

So my original code that used img.ptr(), can work using img.data() like this :

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    //auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    cv::Mat img2(rows, cols, type, (unsigned char*)img.data());
    cv::imshow("test", img2);
}

Turk

Interesting. BTW, that line with img2 seems redungdant here. | I got to the point of calling a pybind wrapped function in embedded Python. It's 3am not tho, so i'll resume tomorrow. Good insights here! – Malleus

Thanks a lot. its really appreciated :) – Turk

Good observation with data rather than ptr. However, keep in mind that right now you're using a C-style cast to cast away const from the data pointer. You could use mutable_data which returns a non-const pointer, and you don't need any cast at all in that case. | Here is a draft of a fairly generic conversion function: pastebin.com/sawt2EFt It still needs some more testing and bullet-proofing tho. – Malleus

Thanks a lot. I'll look at it. really appreciate you kind help and time on this :) – Turk

@DanMašek Do you happen to know how we can expose py::object in C ? consider this function that we have above, it accepts a py::array_t, but if I want to expose this in C, it seems void* doesnt work (on python part it raises the exception). How should we go about this? – Turk

Answer 2

To convert between cv::Mat and np.ndarray, you can use pybind11_opencv_numpy.

Copy ndarray_converter.h and ndarray_converter.cpp to your project directory.

---

CMakeLists.txt

add_subdirectory(pybind11)
execute_process(COMMAND ${PYTHON_EXECUTABLE} -c "import numpy; print(numpy.get_include())" OUTPUT_VARIABLE NUMPY_INCLUDE OUTPUT_STRIP_TRAILING_WHITESPACE)
message(STATUS "NUMPY_INCLUDE: " ${NUMPY_INCLUDE})
include_directories(${NUMPY_INCLUDE})
pybind11_add_module(mymodule "cpp2py.cpp" "ndarray_converter.cpp")
target_link_libraries(mymodule PRIVATE ${OpenCV_LIBS})
target_compile_definitions(mymodule PRIVATE)

---

cpp2py.cpp

#include "ndarray_converter.h"

PYBIND11_MODULE(mymodule, m)
{
    NDArrayConverter::init_numpy();
    ...
}

Staggs

Works simple and perfectly. I just lost some time because i forgot to install numpy correctly with "sudo apt-get install python-numpy" instead of only "pip install numpy". – Lamprophyre

Also, to access object attribute from python side, it is necessary to use: .def_readwrite("frame", &MyCppObject::frame) – Lamprophyre

@Guinther CMake tries to find NumPy library which should have a C++ header, so pip install does not work. I assume you have a class MyCppObject which has a public member cv::Mat frame. Feel free to improve the post by providing a minimal working example. – Staggs

Answer 3

This would be a generic conversion of an image with any number of channels and stride possibly different from the standard one (for example if the Mat has been obtained as a region of interest in a bigger matrix)

#include <pybind11/pybind11.h>

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    cv::Mat mat(img.shape(0), img.shape(1), CV_MAKETYPE(CV_8U, img.shape(2)),
                const_cast<uint8_t*>(img.data()), img.strides(0));

    cv::imshow("test", mat);
}

• img.shape(0) -> rows
• img.shape(1) -> cols
• img.shape(2) -> n_channels
• img.strides(0) -> stride in bytes between two neighboring pixels on the same image column

Ray

Great thanks. It does not work if image.ndim == 2, in which case you have to replace img.shape(2) with 1 to prevent an error. – Plucky

Answer 4

You may also try https://github.com/pthom/cvnp

It provides automatic casts:

• Casts with shared memory between cv::Mat, cv::Matx, cv::Vec and numpy.ndarray
• Casts without shared memory for simple types, between cv::Size, cv::Point, cv::Point3 and python tuple

It also provides explicit transformers between cv::Mat, cv::Matx and numpy.ndarray with shared memory

Freemanfreemartin