C-extension in Python - return Py_BuildValue() memory leak problem

Question

I have a huge memory leak problem involving a C-extension I'm developing. In C, I have an array of doubles called A and an int variable called AnotherIntVariablethat I want to pass to Python. Well, in my C-extension module I do the following:

int i;  
PyObject *lst = PyList_New(len_A);  
PyObject *num;  
if(!lst)  
   return NULL;  
for(i=0;i<len_A;i++){  
   num=PyFloat_FromDouble(A[i]);  
   if(!num){  
      Py_DECREF(lst);  
      return NuLL;  
   }  
   PyList_SET_ITEM(lst,i,num);  
}  
free(A);  
return Py_BuildValue("Oi",lst,AnotherIntVariable)

So in Python i recieve this list and the int like this:

Pyt_A,Pyt_int=MyCModule.MyCFunction(...)

Where Pyt_A and Pyt_int are the list and the integer I get from my C-extension "MyCModule", from the function "MyCFunction" that I described earlier.

The problem is that, in Python, I use this Pyt_A array (so that's why I use Py_BuildValue instead of a simple return statement, to do an INCREF in order to save this variable for a moment from the garbage collector) but then I need to dereference it somehow in order to free that allocated memory. The problem is that I use the MyCFunction function several times, and this produces a memory leakage because I don't know how to dereference the array that I get in python in order to get rid of it.

I tried just returning the array by doing a return lst in the C part of the code instead of the Py_BuildValue("Oi",lst,AnotherIntVariable), but that only results in a Segmentation Fault when I try to use it in python (probably because the garbage collector did his work)...

...what am I missing here? Can anybody help me?

Vullo

doesn't del Pyt_A in python DECREF the array and delete it? – Hebbel

I tried it but it doesn't seem to work. I still get this memory leakage I'm talking about. – Winkle

Also, you do have a return before Py_BuildValue("Oi",lst,AnotherIntVariable) right? – Hebbel

Yup, I missed it on the post editing – Winkle

what is sys.getrefcount(Pyt_A) in python? – Hebbel

The value of sys.getrefcount(Pyt_A) is 3. – Winkle

Answer 1

If you look at the documentation for Py_BuildValue (http://docs.python.org/3/c-api/arg.html#c.Py_BuildValue) you can see that under the O typecode, it says that the reference count of the passed in object is incremented by one (Note: an earlier section in that page describes the O typecode for PyArg_ParseTuple, which doesn't increment the reference count, but also isn't relevant here).

So, after the call to Py_BuildValue, the refcount for your list is 2, but you only want it to be 1.

Instead of returning the result of Py_BuildValue directly, save it to a PyObject pointer, decrement the lst reference count, then return your result.

You should be checking the result of the Py_BuildValue call anyway, since you also need to free num in the event that Py_BuildValue fails (i.e. returns NULL).

Mathias

The documentation now says "The object’s reference count is not increased" - so which is it? – Burkhard

The target link anchor for Py_BuildValue changed, so the link was going to the top of the page, which covers the PyArg_ParseTuple format codes. I fixed the link, and added a note about the potentially confusing entry. – Mathias

Answer 2

Thanks for clearing it up Ignacio, now it makes so much sense! Finally, the solution was to, instead of returning directly the Py_BuildValue, do:

free(A);  
PyObject *MyResult = Py_BuildValue("Oi",lst,AnotherIntVariable);  
Py_DECREF(lst);  
return MyResult

It worked like a charm!

Vullo

Answer 3

There's a simpler solution that what the other answers have been suggesting. Don't rearrange your code to decref after using the O format code (which takes a new strong reference to the passed object), just use the N format code, which steals your reference, avoiding unnecessary reference count manipulation entirely by transferring ownership. The only change needed for this is to change:

return Py_BuildValue("Oi",lst,AnotherIntVariable);

to:

return Py_BuildValue("Ni",lst,AnotherIntVariable);
//                    ^ only change

Py_BuildValue is pretty robust too; if it fails for any reason it continues parsing arguments to ensure reference counts are stolen and released appropriately. If the N argument (lst) is passed as NULL (usually because a function returning a new strong reference was called in the argument list and failed, setting an exception) it preserves that exception and returns NULL itself so the exception propagates as intended.

Bunyan