segmentation fault - core dump in python C-extension

Question

I am writing a c-extension for python. As you can see below, the aim of the code is to calculate the euclidean-dist of two vectors. the first param n is the dimension of the vectors, the second , the third param is the two list of float.

I call the function in python like this:

import cutil
cutil.c_euclidean_dist(2,[1.0,1,0],[0,0])

and it worked well, return the right result. but if i do it for more than 100 times(the dimension is 1*1000), it will cause segmentation fault - core dump:

#!/usr/bin/env python
#coding:utf-8
import cutil
import science
import time
a = []
b = []
d = 0.0 
for x in range(2500):
    a.append([float(i+x) for i in range(1000)])
    b.append([float(i-x) for i in range(1000)])

t1 = time.time()
for x in range(500):
    d += cutil.c_euclidean_dist(1000,a[x],b[x])
print time.time() - t1
print d

the C code is here:

#include <python2.7/Python.h>
#include <math.h>

static PyObject* cutil_euclidean_dist(PyObject* self, PyObject* args) {
    PyObject *seq_a, *seq_b;
    int n;
    float * array_a,* array_b;
    PyObject *item;

    PyArg_ParseTuple(args,"iOO", &n , &seq_a, &seq_b);
    if (!PySequence_Check(seq_a) || !PySequence_Check(seq_b)) {
       PyErr_SetString(PyExc_TypeError, "expected sequence");
       return NULL;
    }

    array_a =(float *)malloc(sizeof(float)*n);
    array_b =(float *)malloc(sizeof(float)*n);  

    if (NULL == array_a || NULL == array_b){
        PyErr_SetString(PyExc_TypeError, "malloc failed!");
        Py_DECREF(seq_a);
        Py_DECREF(seq_b);
        return NULL;
    }

    int i;
    for(i=0;i<n;i++){
        item = PySequence_GetItem(seq_a,i);

        if (!PyFloat_Check(item)) {
            free(array_a);  /* free up the memory before leaving */
            free(array_b);
            Py_DECREF(seq_a);
            Py_DECREF(seq_b);
            Py_DECREF(item);
            PyErr_SetString(PyExc_TypeError, "expected sequence of float");
            return NULL;
        }
        array_a[i] = PyFloat_AsDouble(item);

        Py_DECREF(item);

        item = PySequence_GetItem(seq_b,i);
        if(!PyFloat_Check(item)) {
            free(array_a);
            free(array_b);
            Py_DECREF(seq_a);
            Py_DECREF(seq_b);
            Py_DECREF(item);
            PyErr_SetString(PyExc_TypeError, "expected sequence of float"); 
            return NULL;
        }
        array_b[i] = PyFloat_AsDouble(item);
        Py_DECREF(item);
    }

    double sum = 0;
    for(i=0;i<n;i++){
        double delta = array_a[i] - array_b[i];
        sum += delta * delta;
    }

    free(array_a);
    free(array_b);
    Py_DECREF(seq_a);
    Py_DECREF(seq_b);

    return Py_BuildValue("d",sqrt(sum));
}

static PyMethodDef cutil_methods[] = {
    {"c_euclidean_dist",(PyCFunction)cutil_euclidean_dist,METH_VARARGS,NULL},
    {NULL,NULL,0,NULL}
};

PyMODINIT_FUNC initcutil(void) {
    Py_InitModule3("cutil", cutil_methods, "liurui's c extension for python");
} 

the error msg:

segmentation fault - core dump:

The c-extension is compiled to cutil.so, I do not know how to see the dump. But i looked through my C code for many times,and can not find any problem..

May it be a memory problem?

It should be a very simple piece of C code, what's wrong with it? I need your help~ thanks a lot !

here is the result of gdb /usr/bin/python2.7 ./core:

root@ubuntu:/home/rrg/workspace/opencvTest/test# gdb /usr/bin/python2.7 ./core 
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2.7...Reading symbols from /usr/lib/debug//usr/bin/python2.7...done.
done.

warning: core file may not match specified executable file.
[New LWP 13787]
[New LWP 13789]
[New LWP 13790]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python py.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000005398b3 in list_dealloc.16846 (op=0x7f688b2faa28) at ../Objects/listobject.c:309
309 ../Objects/listobject.c: no such file or directory
#0  0x00000000005398b3 in list_dealloc.16846 (op=0x7f688b2faa28) at ../Objects/listobject.c:309
#1  0x00000000004fdb96 in insertdict_by_entry (value=<optimized out>, ep=0x1777fa8, hash=<optimized out>, key='b', mp=0x7f68a8dbb168) at ../Objects/dictobject.c:519
#2  insertdict (value=<optimized out>, hash=<optimized out>, key='b', mp=0x7f68a8dbb168) at ../Objects/dictobject.c:556
#3  dict_set_item_by_hash_or_entry (value=<optimized out>, ep=0x0, hash=<optimized out>, key='b', 
    op={'a': None, 'x': None, 'c': None, 'b': None, 'd': <float at remote 0x4480b30>, '__builtins__': <module at remote 0x7f68a8de6b08>, 'science': <module at remote 0x7f68a8ce4088>, '__package__': None, 'i': 999, 'cutil': <module at remote 0x7f68a8cdfbb0>, 'time': <module at remote 0x7f68a640ea28>, '__name__': '__main__', 't1': <float at remote 0xd012708>, '__doc__': None}) at ../Objects/dictobject.c:765
#4  PyDict_SetItem (
    op=op@entry={'a': None, 'x': None, 'c': None, 'b': None, 'd': <float at remote 0x4480b30>, '__builtins__': <module at remote 0x7f68a8de6b08>, 'science': <module at remote 0x7f68a8ce4088>, '__package__': None, 'i': 999, 'cutil': <module at remote 0x7f68a8cdfbb0>, 'time': <module at remote 0x7f68a640ea28>, '__name__': '__main__', 't1': <float at remote 0xd012708>, '__doc__': None}, key=key@entry='b', 
    value=<optimized out>) at ../Objects/dictobject.c:818
#5  0x000000000055a9e1 in _PyModule_Clear (m=<optimized out>) at ../Objects/moduleobject.c:139
#6  0x00000000004f2ad4 in PyImport_Cleanup () at ../Python/import.c:473
#7  0x000000000042fa89 in Py_Finalize () at ../Python/pythonrun.c:459
#8  0x000000000046ac10 in Py_Main (argc=<optimized out>, argv=0x7fff3958d058) at ../Modules/main.c:665
#9  0x00007f68a8665ec5 in __libc_start_main (main=0x46ac3f <main>, argc=2, argv=0x7fff3958d058, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff3958d048)
    at libc-start.c:287
#10 0x000000000057497e in _start ()

Edit: I comment the last 2 sentences near the last return:

Py_DECREF(seq_a);
Py_DECREF(seq_b);  

and then it seems to work well. I feel very very strange... The purpose of the two sentence is to free(or release) the two pyobject, why it works well without the two sentences which I think are necessary?

Intercede

Unrelated to your problem, but DO not cast the return value of malloc() and if either one of the malloc()s succeeds you will have a memory leak in ` if (NULL == array_a || NULL == array_b)` – Domineering

Why are you decreffing the items once you get them, does the documentation says you have to? – Domineering

@iharob for the first problem, i got it and will not cast the pointer from malloc(). – Intercede

@iharob the second one, the doc does not ask me to do like this, i did it by myself becasue I think if i do not use decref() at once , there will be a memory leak after the "item" is set to other "object" – Intercede

@plaes could you help me with this? I really need your help ~ – Intercede

Why are you writing a c extension when you can just do it in pure python? is it too slow? – Domineering

yes, the cost of python is 10 times as C according to my test – Intercede

Answer 1

The c-extension is compiled to cutil.so, I do not know how to see the dump.

To solve this, I'm going to cite GNU Radio's GDB/Python debugging mini-tutorial:

Luckily, there's a feature called core dumping that allows the state of your program to be stored in a file, allowing later analysis. Usually, that feature is disabled; you can enable it by:

ulimit -c unlimited

Note that this only works for processes spawned from the shell that you used ulimit in. What happens here is that the maximum size of a core dump is set to unlimited (the original value is 0 in most cases).

Now, the core dump file lays in the current execution directory of the program that crashed. In our case, that's build/python/, but since all core dumps should have a name like core., we can use a little find magic:

marcus> find -type f -cmin 5 -name 'core.[0-9]*'

./build/python/core.22608

because that will find all _f_iles, changed/created within the last _5 min_utes, having a name that matches.

Using GDB with a core dump

having found build/python/core.22608, we can now launch GDB:

gdb programname coredump

i.e.

gdb /usr/bin/python2 build/python/core.22608

A lot of information might scroll by.

At the end, you're greeted by the GDB prompt:

(gdb) 

Getting a backtrace

Typically, you'd just get a backtrace (or shorter, bt). A backtrace is simply the hierarchy of functions that were called.

 (gdb)bt

[...] skipped,

Frame #2 and following definitely look like they're part of the Python implementation -- that sounds bad, because GDB doesn't itself know how to debug python, but luckily, there's an extension to do that. So we can try to use py-bt:

(gdb) py-bt

If we get a undefined command error, we must stop here and make sure that the python development package is installed (python-devel on Redhatoids, python2.7-dev on Debianoids); for some systems, you should append the content of /usr/share/doc/{python-devel,python2.7-dev}/gdbinit[.gz] to your ~/.gdbinit, and re-start gdb.

The output of py-bt now states clearly which python lines correspond to which stack frame (skipping those stack frames that are hidden to python, because they are in external libraries or python-implementation routines)

...

Azevedo

I run gdb /usr/bin/python2.7 ./core , and it gives a warning: core file may not match specified executable file. but it still gives some stack info, i copied the stack info above, can you please help me identify the problem? – Intercede

If you're unsure if you're using the right executable, you could also try running the program by gdb --args python mypythonfile.py as that should ensure the right executable is used – Massa

@user3978288: no problem; though I would be really happy if you upvoted or accepted my answer :) – Massa

Answer 2

thanks for the 2 kind and nice guys above who helped me.

Problem seemed to be solved.

comment the 2 lines:

Py_DECREF(seq_a); 
Py_DECREF(seq_b);

for more details pls read python offical doc on C-API

I guess the reason is that the seq_a seq_b get from argv is a "borrowed reference" rather than a real refference , so we do not need to decref().

but as the official docs says, if you convert the borrowed refference into a real referrence using a incref() , then you should call decref()

U can also search "python c-extension referrence count" for more detail

Intercede