numpy array C api

Question

I have a C++ function returning a std::vector and I want to use it in python, so I'm using the C numpy api:

static PyObject *
py_integrate(PyObject *self, PyObject *args){
    ...
    std::vector<double> integral;
    cpp_function(integral);  // This changes integral
    npy_intp size = {integral.size()};
    PyObject *out = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, &(integral[0]));
    return out;
}

Here's how I call it from python:

import matplotlib.pyplot as plt

a = py_integrate(parameters)
print a
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(a)
print a

What happens is: The first print is ok, the values are correct. But when I plot a they are not; in the second print I see very strange values like 1E-308 1E-308 ... or 0 0 0 ... as an uninitialized memory. I don't understand why the first print is ok.

Partial solution (not working):

static void DeleteVector(void *ptr)
{
    std::cout << "Delete" << std::endl;
    vector * v = static_cast<std::vector<double> * >(ptr);
    delete v;
    return;
}

static PyObject *
cppfunction(PyObject *self, PyObject *args)
{
    std::vector<double> *vector = new std::vector<double>();
    vector->push_back(1.);
    PyObject *py_integral = PyCObject_FromVoidPtr(vector, DeleteVector);
    npy_intp size = {vector->size()};
    PyArrayObject *out;
    ((PyArrayObject*) out)->base = py_integral;
    return (PyObject*)(out);
}

Holds

Answer 1

Your std::vector object appears to be local to that function. PyArray_SimpleNewFromData does not make a copy of the data you pass it. It just keeps a pointer. So once your py_integrate function returns, the vector is deallocated. The print works the first time because nothing has written over the freed memory yet, but by the time you get to the next print, something else has used that memory, causing the values to be different.

You need to make a NumPy array that owns its own storage space and then copy the data into it.

Alternatively, allocate your vector on the heap. Then store a pointer to it in a CObject. Provide a destructor that deletes the vector. Then, take a look at the C-level PyArrayObject type. It has a PyObject * member called base. Store your CObject there. Then when the NumPy array is garbage collected, the reference count on this base object will be decremented, and assuming you haven't taken a copy of it elsewhere, your vector will be deleted thanks to the destructor you provided.

Fixer-upper

You forgot to actually create the PyArray. Try this:

(You didn't post DeleteVector, so I can only hope that it's right)

std::vector<double> *vector = new std::vector<double>();
vector->push_back(1.);
PyObject *py_integral = PyCObject_FromVoidPtr(vector, DeleteVector);
npy_intp size = {vector->size()};
PyObject *out = PyArray_SimpleNewFromData(1, &size, NPY_DOUBLE, &((*vector)[0]));
((PyArrayObject*) out)->base = py_integral;
return out;

Note: I'm not a C++ programmer, so I can only assume that &((*vector)[0]) works as intended with a pointer to a vector. I do know that the vector reallocate its storage area if you grow it, so don't increase its size after getting that pointer or it won't be valid anymore.

Calendre

thanks, what does it means "provide a destructor that deletes the vector"? std::vector already has its destructor. – Holds

@wiso - Yes, std::vector has a destructor, but the Python CObject does not, and that is what is responsible for deleting the vector. All you need is to provide a small function that deletes the vector contained within it. – Calendre

Python is written in C. The CObject is Python's way of encapsulating a pointer to some arbitrary object as a Python object. You need Python to keep track of your vector so that it can be deleted after the numpy array. However, Python does not understand C++, so you must tell it how to delete the vector. This is in the link to the CObject documentation I provided. If you can't figure this out, just use the first option instead. – Calendre

I got Segmentation fault, can you be more specific on how to construct PyArrayObject, I simply do: PyArrayObject *out = new PyArrayObject(); out->base = py_integral; where py_integral is a PyObject constructed by PyCObject_FromVoidPtr – Holds

You don't need to construct a new PyArrayObject if you're using the alternative method (and if you did, you shouldn't do it that way, since it's not a C++ class). Just cast the PyObject that you had already. I think it should probably go something like ((PyArrayObject*) out)->base = py_integral; – Calendre

sorry kwatford, can you fix my code (the not working solution in the question)? – Holds

Thanks a lot, it works. Last question: I put a std::cout inside the DeleteVector to check if it is called when I delete the python object. It works, but the problem is this: a = cfunction(); b = a; del a and I see that the DeleteVector is call, but there is a refence to the object! And I can't do print b. For example a = [1,2,3]; b=a; del a the content of b is not deleted. – Holds

I would need to see more of your code. The situation you are describing sounds strange, since if you do b = a;, b is a. It is possible that you have an extra decref or not enough increfs somewhere. – Calendre

all my code is in the question. As you can see I don't manage ref counter – Holds

Hmm. Well, from what you've posted, I can't determine what is wrong. But it looks like you typed that code (I can tell by the doubble typo), so you may have unintentionally left something out. You could copy & paste all of it, including the modifications, for a more accurate reproduction. – Calendre

Answer 2

You will need to make a copy of the vector, since the vector will go out of scope and the memory will no longer be usable by the time you need it in Python (as stated by kwatford).

One way to make the Numpy array you need (by copying the data) is:

PyObject *out = nullptr;

std::vector<double> *vector = new std::vector<double>();
vector->push_back(1.);

npy_intp size = {vector.size()};

out = PyArray_SimpleNew(1, &size, NPY_DOUBLE);

memcpy(PyArray_DATA((PyArrayObject *) out), vector.data(), vector.size());

Mcnulty