Sending a C++ array to Python and back (Extending C++ with Numpy)

Question

I am going to send a c++ array to a python function as numpy array and get back another numpy array. After consulting with numpy documentation and some other threads and tweaking the code, finally the code is working but I would like to know if this code is written optimally considering the:

• Unnecessary copying of the array between c++ and numpy (python).
• Correct dereferencing of the variables.
• Easy straight-forward approach.

C++ code:

// python_embed.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "Python.h"
#include "numpy/arrayobject.h"
#include<iostream>

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    import_array()

    // Build the 2D array
    PyObject *pArgs, *pReturn, *pModule, *pFunc;
    PyArrayObject *np_ret, *np_arg;
    const int SIZE{ 10 };
    npy_intp dims[2]{SIZE, SIZE};
    const int ND{ 2 };
    long double(*c_arr)[SIZE]{ new long double[SIZE][SIZE] };
    long double* c_out;
    for (int i{}; i < SIZE; i++)
        for (int j{}; j < SIZE; j++)
            c_arr[i][j] = i * SIZE + j;

    np_arg = reinterpret_cast<PyArrayObject*>(PyArray_SimpleNewFromData(ND, dims, NPY_LONGDOUBLE, 
        reinterpret_cast<void*>(c_arr)));

    // Calling array_tutorial from mymodule
    PyObject *pName = PyUnicode_FromString("mymodule");
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (!pModule){
        cout << "mymodule can not be imported" << endl;
        Py_DECREF(np_arg);
        delete[] c_arr;
        return 1;
    }
    pFunc = PyObject_GetAttrString(pModule, "array_tutorial");
    if (!pFunc || !PyCallable_Check(pFunc)){
        Py_DECREF(pModule);
        Py_XDECREF(pFunc);
        Py_DECREF(np_arg);
        delete[] c_arr;
        cout << "array_tutorial is null or not callable" << endl;
        return 1;
    }
    pArgs = PyTuple_New(1);
    PyTuple_SetItem(pArgs, 0, reinterpret_cast<PyObject*>(np_arg));
    pReturn = PyObject_CallObject(pFunc, pArgs);
    np_ret = reinterpret_cast<PyArrayObject*>(pReturn);
    if (PyArray_NDIM(np_ret) != ND - 1){ // row[0] is returned
        cout << "Function returned with wrong dimension" << endl;
        Py_DECREF(pFunc);
        Py_DECREF(pModule);
        Py_DECREF(np_arg);
        Py_DECREF(np_ret);
        delete[] c_arr;
        return 1;
    }
    int len{ PyArray_SHAPE(np_ret)[0] };
    c_out = reinterpret_cast<long double*>(PyArray_DATA(np_ret));
    cout << "Printing output array" << endl;
    for (int i{}; i < len; i++)
        cout << c_out[i] << ' ';
    cout << endl;

    // Finalizing
    Py_DECREF(pFunc);
    Py_DECREF(pModule);
    Py_DECREF(np_arg);
    Py_DECREF(np_ret);
    delete[] c_arr;
    Py_Finalize();
    return 0;
}

In CodeReview, there is a fantastic answer: Link...

Lorrinelorry

have you looked at Boost.Numpy? See this very simple example: github.com/ndarray/Boost.NumPy/blob/master/libs/numpy/example/… – Deprecatory

or using Cython? https://mcmap.net/q/374762/-passing-and-returning-numpy-arrays-to-c-methods-via-cython – Deprecatory

@denfromufa, I don't want to use boost here, also cython is not an option as I am extending C++ using python/numpy. – Lorrinelorry

Did you look at SWIG (docs.scipy.org/doc/numpy/reference/swig.interface-file.html)? It allows to have memory views in numpy on C++-arrays and vice versa. It takes care of the wrapping automatically. Though I have never used it in the C++ to Python direction. – Shantell

@Dietrich, Thanks, SWIG is also good, but here I would like to work purely with Python/C++ API. I appreciate any comments on the provided code. – Lorrinelorry

I'd say that looks pretty good, however your question would be a better fit for Code Review, since your code is already working :) – Epiphenomenon

What about the python code? Can you show that as well, in order to test? – Cambodia

It is provided in code review – Lorrinelorry

Answer 1

Try out xtensor and the xtensor-python python bindings.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

• an extensible expression system enabling numpy-style broadcasting (see the numpy to xtensor cheat sheet).
• an API following the idioms of the C++ standard library.
• tools to manipulate array expressions and build upon xtensor.
• bindings for Python, but also R and Julia.

Example of usage

Initialize a 2-D array and compute the sum of one of its rows and a 1-D array.

#include <iostream>
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray<double> arr1
  {{1.0, 2.0, 3.0},
   {2.0, 5.0, 7.0},
   {2.0, 5.0, 7.0}};

xt::xarray<double> arr2
  {5.0, 6.0, 7.0};

xt::xarray<double> res = xt::view(arr1, 1) + arr2;

std::cout << res;

Outputs

{7, 11, 14}

Creating a Numpy-style universal function in C++.

#include "pybind11/pybind11.h"
#include "xtensor-python/pyvectorize.hpp"
#include <numeric>
#include <cmath>

namespace py = pybind11;

double scalar_func(double i, double j)
{
    return std::sin(i) - std::cos(j);
}

PYBIND11_PLUGIN(xtensor_python_test)
{
    py::module m("xtensor_python_test", "Test module for xtensor python bindings");

    m.def("vectorized_func", xt::pyvectorize(scalar_func), "");

    return m.ptr();
}

Python code:

import numpy as np
import xtensor_python_test as xt

x = np.arange(15).reshape(3, 5)
y = [1, 2, 3, 4, 5]
z = xt.vectorized_func(x, y)
z

Outputs

[[-0.540302,  1.257618,  1.89929 ,  0.794764, -1.040465],
 [-1.499227,  0.136731,  1.646979,  1.643002,  0.128456],
 [-1.084323, -0.583843,  0.45342 ,  1.073811,  0.706945]]

Monarchism

seems interesting, but you are extending python using C++, I need to extend C++ using python. – Lorrinelorry

Ok this actually something that is done in the tests for xtensor where we instanciate a python interpreter from c++ and start creating arrays. – Monarchism

Answer 2

We will be passing 2D array to python function written in file pyCode.py:

def pyArray (a):
    print ("Contents of a :")
    print (a)
    c = 0
    return c

1. For C++ to Python: File: c_code.cpp

#include <Python.h>
#include <stdio.h>
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <numpy/arrayobject.h>

float Array [] = {1.2, 3.4, 5.6, 7.8};

int main (int argc, char *argv[])
{
    float *ptr = Array;
    PyObject *pName, *pModule, *pDict, *pFunc, *pArgs;
    npy_intp dims[1] = { 4 };
    PyObject *py_array;

    setenv("PYTHONPATH",".",1);
    Py_Initialize ();
    pName = PyUnicode_FromString ("pyCode");

    pModule = PyImport_Import(pName);

    pDict = PyModule_GetDict(pModule);

    import_array ();                                   

    py_array = PyArray_SimpleNewFromData(1, dims, NPY_FLOAT, ptr);
    

    pArgs = PyTuple_New (1);
    PyTuple_SetItem (pArgs, 0, py_array);

    pFunc = PyDict_GetItemString (pDict, (char*)"pyArray"); 

    if (PyCallable_Check (pFunc))
    {
        PyObject_CallObject(pFunc, pArgs);
    } else
    {
        cout << "Function is not callable !" << endl;
    }

    Py_DECREF(pName);
    Py_DECREF (py_array);                             
    Py_DECREF (pModule);
    Py_DECREF (pDict);
    Py_DECREF (pFunc);

    Py_Finalize ();                                    

    return 0;
}

compile the code: g++ -g -fPIC c_code.cpp -o runMe -lpython3.5m -I/usr/include/python3.5m/

2. From OpenCV Mat to Python:

file: cv_mat_code.cpp

#include <iostream>
#include <Python.h>
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <numpy/arrayobject.h>

#include <opencv2/opencv.hpp>

using namespace cv;
using namespace std;

int main (int argc, char *argv[])
{
    float data[10] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

    Mat mat1 (cv::Size (5, 2), CV_32F, data, Mat::AUTO_STEP);
    int row = 0;
    float *p = mat1.ptr<float>(row);

    cout << "Mat" << mat1 <<endl;

    PyObject *pName, *pModule, *pDict, *pFunc, *pArgs;
    npy_intp dims[2] = { 2, 5 };
    PyObject *py_array;

    setenv("PYTHONPATH",".",1);
    Py_Initialize ();
    pName = PyUnicode_FromString ("pyCode");
    
    pModule = PyImport_Import(pName);

    pDict = PyModule_GetDict(pModule);

    // Required for the C-API : http://docs.scipy.org/doc/numpy/reference/c-api.array.html#importing-the-api
    import_array ();

    py_array = PyArray_SimpleNewFromData(2, dims, NPY_FLOAT, p);

    pArgs = PyTuple_New (1);
    PyTuple_SetItem (pArgs, 0, py_array);

    pFunc = PyDict_GetItemString (pDict, (char*)"pyArray"); 

    if (PyCallable_Check (pFunc))
    {
        PyObject_CallObject(pFunc, pArgs);
    } else
    {
        cout << "Function is not callable !" << endl;
    }

    Py_DECREF(pName);
    Py_DECREF (py_array);                             
    Py_DECREF (pModule);
    Py_DECREF (pDict);
    Py_DECREF (pFunc);

    Py_Finalize ();                                  

    return 0;
}

Compile the code: g++ -g -fPIC cv_mat_code.cpp -o runMe -lpython3.5m -I/usr/include/python3.5m/ -I/usr/include/ -lopencv_core -lopencv_imgproc -lopencv_highgui

Conceit

why do you have 42 float in your data array but assign only 10 elements in your Mat? – Fitter

@Fitter Thanks! made the changes. – Conceit

The PyObject_callObject gets blocked if there's a cv2 call inside there. Is this a known issue with opencv for python 2.7 – Fitter

its called python code embedding and hence they are blocking. To understand how main process invokes interpreter here. – Conceit

I misspoke, what I meant to say is it hangs at the cv2 call and doesn't proceed. – Fitter

good answer but doesn't return numpy array back to c++ – Fitter

In the first C++ code, the iostream module is missing because your are using cout and endl. No need to import stdio.h – Butterfly

Answer 3

As an additional way, without touching directly to the Python C API, it is possible to use pybind11 ( header-only library) :

CPP :

#include <pybind11/embed.h> // everything needed for embedding
#include <iostream>
#include <Eigen/Dense>  
#include<pybind11/eigen.h>
using Eigen::MatrixXd;
namespace py = pybind11;

int main() 
{    
  try 
  {          
        Py_SetProgramName("PYTHON");
        py::scoped_interpreter guard{}; 

        py::module py_test = py::module::import("py_test");

        MatrixXd m(2,2);
        m(0,0) = 1;
        m(1,0) = 2;
        m(0,1) = 3;
        m(1,1) = 4;

        py::object result = py_test.attr("test_mat")(m);

        MatrixXd res = result.cast<MatrixXd>();
        std::cout << "In c++ \n" << res << std::endl;
  }
  catch (std::exception ex)
  {
      std::cout << "ERROR   : " << ex.what() << std::endl;
  }
  return 1;
}

In py_test.py :

def test_mat(m):
    print ("Inside python m = \n ",m )
    m[0,0] = 10
    m[1,1] = 99 
    return m

Output :

Inside python m =
  [[ 1.  3.]
  [ 2.  4.]]
In c++
10  3
 2 99

See the official documentation.

ps: I'm using Eigen for the C++ Matrix.

Sible

Thanks for the great example; I've just replicated it. Found that this line needed to be: std::cout << "In c++ \n" << res << std::endl; – Garcia

Answer 4

From my experience that seems to be pretty efficient. To get even more efficiency out of it try this : http://ubuntuforums.org/showthread.php?t=1266059

Using weave you can inline C/C++ code in Python so that could be useful.

http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.weave.inline.html

Here's a link on how Python can be used to interface between many different languages along with examples.

http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html

This is a quick and easy example of how to pass numpy arrays to c++ using Cython:

http://www.birving.com/blog/2014/05/13/passing-numpy-arrays-between-python-and/

Rhu

I am extending c++ using python not the opposite so cython, weave are not relevant. Only your first link might be relevant. I appreciate if you could put any possible improvement to the provided code though. – Lorrinelorry

Cython is still relevant as you can call it from C++/C by using the public keyword. – Dactylogram

@BeRecursive, yes it could also work. Now I am mainly concerned about the performance of my code given in the question. – Lorrinelorry

If you want to improve the speed of the Python bit I can also suggest using pypy.org Using PyPy increases the speed drastically. – Rhu

Weave is pretty much deprecated in favor of Cython now. The standalone weave package recommends using it only if you have existing code that depends on weave which has not yet been ported. – Vitek