python tracing a segmentation fault

Question

I'm developing C extensions from python and I obtain some segfaults (inevitable during the development...).

I'm searching for a way to display at which line of code the segfault happens (an idea is like tracing every single line of code), how can I do that?

Aleda

Answer 1

Here's a way to output the filename and line number of every line of Python your code runs:

import sys

def trace(frame, event, arg):
    print("%s, %s:%d" % (event, frame.f_code.co_filename, frame.f_lineno))
    return trace

def test():
    print("Line 8")
    print("Line 9")

sys.settrace(trace)
test()

Output:

call, test.py:7
line, test.py:8
Line 8
line, test.py:9
Line 9
return, test.py:9

(You'd probably want to write the trace output to a file, of course.)

Grolier

@MadPhysicist: It won't print the line numbers of your C code, if that's what you mean. :-) It will print the line numbers of the Python code that calls into your C code. – Grolier

That was what I meant. I found the original question interesting because I had the same problem. The segfault turned out to be because my C code was inserting a NULL element into a PyList_Object. It manifested itself on the Python side when I tried to iterate over the list. Not sure a python debugger would have helped much in that case. – Cockahoop

To make the output even easier to read (and it won't slow down the program significantly), use the faulthandler module mentioned in an answer below) -- but even if you want to use trace, just use Python's built-in trace module (python -m trace --trace test.py) -- both approaches won't show a traceback in the C code, however. – Otherworld

I would like to add that I just applied this to my own code for debugging. It appears that the sys.settrace option is active permanently until you reset it with sys.settrace(None). Please do keep this in mind or you will have problems with future python script execution and will see your code be super slow etc. – Aminoplast

Answer 2

If you are on linux, run python under gdb

gdb python
(gdb) run /path/to/script.py
## wait for segfault ##
(gdb) backtrace
## stack trace of the c code

Filtration

If you've got a core file already, you can use gdb python core (or whatever the core file is called). If you're on OSX, core dumps (not generated by default; see ulimit -c) are stored in the directory /cores. – Valeta

I really wish this was the first answer for me as reading all the others took considerable time I'd rather have spent running down my bug. – Conscription

If you get a segfault while running a Python unit test like python -m unittest my.module.tests.mytest, then the -m switch confuses gdb. Invoke using the --args option thusly: gdb --args python -m unittest my.module.tests.mytest – Atmospheric

Similarly with a command: `gdb --args python3.6 -c "import something; something.doStuff();" @Filtration your answer has been really helpful to me, thanks! – Barbrabarbuda

It just says "no executable file specified" – Sebi

This is great. I was wondering how to run my server in docker directly without stepping into the gdb repl. This is how you do that: gdb -ex r --args python <programname>.py <arguments> – Dress

Answer 3

Segfaults from C extensions are very frequently a result of not incrementing a reference count when you create a new reference to an object. That makes them very hard to track down as the segfault occurs only after the last reference is removed from the object, and even then often only when some other object is being allocated.

You don't say how much C extension code you have written so far, but if you're just starting out consider whether you can use either ctypes or Cython. Ctypes may not be flexible enough for your needs, but you should be able to link to just about any C library with Cython and have all the reference counts maintained for you automatically.

That isn't always sufficient: if your Python objects and any underlying C objects have different lifetimes you can still get problems, but it does simplify things considerably.

Standpoint

Also, putting NULLs in places where they don't belong. – Cockahoop

Answer 4

I came here looking for a solution to the same problem, and none of the other answers helped me. What did help was faulthandler, and you can install it in Python 2.7 just using pip install.

faulthandler was introduced to Python only in version 3.3, that was released in September 2012, which was after most other answers here were written.

Dressler

Nice! If you can read stdout of your Python program, usage is as easy as: import faulthandler; faulthandler.enable() – Loppy

Answer 5

There are somewhat undocumented python extensions for gdb.

From the Python source grab Tools/gdb/libpython.py (it is not included in a normal install).

Put this in sys.path

Then:

# gdb /gps/python2.7_x64/bin/python coredump
...
Core was generated by `/usr/bin/python script.py'.
Program terminated with signal 11, Segmentation fault.
#0  call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
...
(gdb) python
>import libpython
>
>end
(gdb) bt
#0  call_function (oparg=<optimized out>, pp_stack=0x7f9084d15dc0) at Python/ceval.c:4037
#1  PyEval_EvalFrameEx (f=f@entry=
    Frame 0x7f9084d20ad0, 
    for file /usr/lib/python2.7/site-packages/librabbitmq/__init__.py, line 220, 
    in drain_events (self=<Connection(channels={1: <Channel(channel_id=1, connection=<...>, is_open=True, connect_timeout=4, _default_channel=<....(truncated), throwflag=throwflag@entry=0) at Python/ceval.c:2681
...
(gdb) py-list
 218            else:
 219                timeout = float(timeout)
>220            self._basic_recv(timeout)
 221
 222        def channel(self, channel_id=None):

As you can see we now have visibility into the Python stack corresponding with the CPython call chain.

Some caveats:

• Your version of gdb needs to be greater than 7 and it needs to have been compiled with --with-python
• gdb embeds python (by linking to libpython), it doesn't run it in a subshell. This means that It may not necessarily match the version of python that is on $PATH.
• You need to download libpython.py from whatever version of the Python source that matches whatever gdb is linked to.
• You may have to run gdb as root - if so you may need to set up sys.path to match that of the code that you are debugging.

If you cannot copy libpython.py into sys.path then you can add it's location to sys.path like this:

(gdb) python
>import sys
>sys.path.append('/path/to/containing/dir/')
>import libpython
>
>end

This is somewhat poorly documented in the python dev docs, the fedora wiki and the python wiki

If you have an older gdb or just can't get this working there is also a gdbinit in the Python source that you can copy to ~/.gdbinit which add some similar functionality

Stockman

Alternatively, one can start a script like this: gdb --args python3 script.py and then execute the run command from the gdb-shell. After the script crashed, one can execute the backtrace command to get the trace. – Trouveur

Answer 6

Here are 3 more alternatives:

1: Executing a script with faulthandler enabled:

python3 -X faulthandler your_script.py

2: Executing a script in debug mode (pdb)

python3 -m pdb your_script.py

and execute the script with the continue command.

The gdb tool provided the most information, but none of them print the last executed line number in my script.

3: I ended up using pytest. For this to work, I wrapped my code in a function prefixed with test_ and execute the script like this:

pytest your_script.py

Trouveur

Answer 7

Mark's answer is awesome. If you happen to be on a machine that has lldb readily available in your path, and not gdb (which was my case), Mark's answer becomes:

lldb python
(lldb) process launch -- /path/to/script.py
## wait for segfault ##
(lldb) bt
## stack trace of the c code

I wish I would have stumbled on this answer earlier :)

Cliff