Showing the stack trace from a running Python application
Asked Answered
J

29

399

I have this Python application that gets stuck from time to time and I can't find out where.

Is there any way to signal Python interpreter to show you the exact code that's running?

Some kind of on-the-fly stacktrace?

Related questions:

Jilolo answered 25/9, 2008 at 8:6 Comment(3)
related: https://mcmap.net/q/25091/-python-is-it-possible-to-attach-a-console-into-a-running-process/1449460Bertiebertila
related wiki.python.org/moin/DebuggingWithGdbArnulfo
This question debugging - Get stacktrace from stuck python process - Stack Overflow asks for the case where signals doesn't work (although some other answers below addresses that case too)Vetavetch
R
353

I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons. Its a bit hacky, and only works on unix (requires signals):

import code, traceback, signal

def debug(sig, frame):
    """Interrupt running process, and provide a python prompt for
    interactive debugging."""
    d={'_frame':frame}         # Allow access to frame object.
    d.update(frame.f_globals)  # Unless shadowed by global
    d.update(frame.f_locals)

    i = code.InteractiveConsole(d)
    message  = "Signal received : entering python shell.\nTraceback:\n"
    message += ''.join(traceback.format_stack(frame))
    i.interact(message)

def listen():
    signal.signal(signal.SIGUSR1, debug)  # Register handler

To use, just call the listen() function at some point when your program starts up (You could even stick it in site.py to have all python programs use it), and let it run. At any point, send the process a SIGUSR1 signal, using kill, or in python:

    os.kill(pid, signal.SIGUSR1)

This will cause the program to break to a python console at the point it is currently at, showing you the stack trace, and letting you manipulate the variables. Use control-d (EOF) to continue running (though note that you will probably interrupt any I/O etc at the point you signal, so it isn't fully non-intrusive.

I've another script that does the same thing, except it communicates with the running process through a pipe (to allow for debugging backgrounded processes etc). Its a bit large to post here, but I've added it as a python cookbook recipe.

Randalrandall answered 25/9, 2008 at 13:38 Comment(14)
Thanks! This is just what I was looking for. Maybe you could also post that script with pipe support on some Python snippets site?Jilolo
I've now posted it at the python cookbook site - link added.Randalrandall
I needed to add "import readline" to enable history features.Retrieve
Great tip! This also works to send the signal, to all processes containing the word "mypythonapp": pkill -SIGUSR1 -f mypythonappPocked
Unfortunately this does not work in mod_wsgi: code.google.com/p/modwsgi/wiki/…Stockpile
Awesome tip indeed. I'm debugging a software that just hangs if I use the standard pdb module. I was really running out of options besides print.Semitics
This does not work for Windows! There is no signal.SIGUSR1 available.Pricking
This works on cygwin! Is there a way to resume the program as is from the shell once you've broken into it like this?Tessie
If the application is stuck, the Python interpreter loop may not be able to run to process the signal. Use the faulthandler module (and its backport found on PyPI) for a C level signal handler that'll print the Python stack without requiring the interpreter loop to be responsive.Trimester
This method seems to not work in multithreaded program. Signal is not catched, tried listening for different signals (USR1, USR2, SIGALRM, SIGTERM), no one works. Possible explanation is code.activestate.com/recipes/… , but this provides no mean to get stack trace of program that is stuck. Any ideas?Panzer
This is a shorter version if you need just the stacktrace: signal.signal(signal.SIGUSR1, lambda sig, frame: traceback.print_stack())Astrophysics
ValueError: signal only works in main thread do we something for multithreaded app in pythonUnsecured
@BjornTipling How does PDB do this?Sulphurize
Is there much (or any) overhead to listening for signals like this?Denitrify
O
158

The suggestion to install a signal handler is a good one, and I use it a lot. For example, bzr by default installs a SIGQUIT handler that invokes pdb.set_trace() to immediately drop you into a pdb prompt. (See the bzrlib.breakin module's source for the exact details.) With pdb you can not only get the current stack trace (with the (w)here command) but also inspect variables, etc.

However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in ~/.gdbinit, then:

  • Attach gdb: gdb -p PID
  • Get the python stack trace: pystack

It's not totally reliable unfortunately, but it works most of the time. See also https://wiki.python.org/moin/DebuggingWithGdb

Finally, attaching strace can often give you a good idea what a process is doing.

Onondaga answered 29/9, 2008 at 0:44 Comment(5)
Brilliant! The pystack command locks up sometimes, but before it does it gives me a complete stack trace of the process, in python code lines, without having needed to do any preparation.Jammie
Minor update: this gdb technique (and updated code) is documented at wiki.python.org/moin/DebuggingWithGdb There's been some development on this front, documented at that URL, and apparently gdb 7 has some Python support.Cheese
As far as I can tell, this only really works if you have debug symbols compiled into your python binary - eg: you ran your program with python2-dbg (on Ubuntu, this is in a separate package python-dbg). Without those symbols, you don't seem to get much useful info.Mammoth
in my case this return Unable to locate python frame to each commandMachinegun
gdb 7+ --with-python support is provided by python-gdb.py. More details here: chezsoi.org/lucas/blog/2014/11/07/en-gdb-python-macrosRavelin
W
81

I am almost always dealing with multiple threads and main thread is generally not doing much, so what is most interesting is to dump all the stacks (which is more like the Java's dump). Here is an implementation based on this blog:

import threading, sys, traceback

def dumpstacks(signal, frame):
    id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
    code = []
    for threadId, stack in sys._current_frames().items():
        code.append("\n# Thread: %s(%d)" % (id2name.get(threadId,""), threadId))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    print("\n".join(code))

import signal
signal.signal(signal.SIGQUIT, dumpstacks)
Wandie answered 2/4, 2010 at 23:23 Comment(3)
python 3 has a much simpler solution- call traceback.print_stack()Ammons
Which threads does print_stack print from?Byway
See also stackoverflow.com/a/24334576 for python 3 to print all threads.Byway
G
60

Getting a stack trace of an unprepared python program, running in a stock python without debugging symbols can be done with pyrasite. Worked like a charm for me in on Ubuntu Trusty:

$ sudo pip install pyrasite
$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
$ sudo pyrasite 16262 dump_stacks.py # dumps stacks to stdout/stderr of the python program

(Hat tip to @Albert, whose answer contained a pointer to this, among other tools.)

Greyso answered 26/4, 2015 at 18:22 Comment(6)
This worked great for me, where dump_stacks.py was simply import traceback; traceback.print_stack()Fusillade
traceback -l gives you a list of predefined python scripts you can use, and dump_stacks.py is one of them. If you are using your own (for instance to write stack trace to a file) it might be wise to use a different name.Porcine
Important tip: run apt-get install gdb python-dbg (or equivalent) before running pyrasite, otherwise it will silently fail. Works like a charm otherwise!Porcine
Last release of pyrasite was in 2012Batter
(disclaimer: my package) I have a fork pyrasite-ng which fixes various reported bugs.Vetavetch
Also, pyrasite-shell hung for me.Byway
G
37
>>> import traceback
>>> def x():
>>>    print traceback.extract_stack()

>>> x()
[('<stdin>', 1, '<module>', None), ('<stdin>', 2, 'x', None)]

You can also nicely format the stack trace, see the docs.

Edit: To simulate Java's behavior, as suggested by @Douglas Leeder, add this:

import signal
import traceback

signal.signal(signal.SIGUSR1, lambda sig, stack: traceback.print_stack(stack))

to the startup code in your application. Then you can print the stack by sending SIGUSR1 to the running Python process.

Gudrunguelderrose answered 25/9, 2008 at 8:29 Comment(2)
This would only print the backtrace of main thread. I am yet to find a solution for seeing traces for all threads. In fact, python seems to lack an API to retrieve stack from Thread object, though threading.enumerate() gives access to all Thread objects.Wandie
This works great on cygwin. It only prints three lines of the stack trace though, but that's enough to get a clueTessie
C
30

The traceback module has some nice functions, among them: print_stack:

import traceback

traceback.print_stack()
Carnallite answered 25/9, 2008 at 8:27 Comment(2)
To write out the stack trace to a file use: import traceback; f = open('/tmp/stack-trace.log', 'w') traceback.print_stack(file=f) f.close()Elfredaelfrida
+1 to @Carnallite for his easy to use answer. Some of the other answers looked very complicated for my simple task of getting call-stack trace from a script's function.Elfredaelfrida
P
27

You can try the faulthandler module. Install it using pip install faulthandler and add:

import faulthandler, signal
faulthandler.register(signal.SIGUSR1)

at the beginning of your program. Then send SIGUSR1 to your process (ex: kill -USR1 42) to display the Python traceback of all threads to the standard output. Read the documentation for more options (ex: log into a file) and other ways to display the traceback.

The module is now part of Python 3.3. For Python 2, see http://faulthandler.readthedocs.org/

Piwowar answered 26/4, 2013 at 22:18 Comment(1)
This one worked for me. My app was crashing and I had no obvious idea why. Thanks to faulthandler I was able to identify line of code that was causing a crash. I was trying to change state of several main UI items by just trying to set them from other thread instead using signals to do that in the main thread.Thiouracil
U
20

What really helped me here is spiv's tip (which I would vote up and comment on if I had the reputation points) for getting a stack trace out of an unprepared Python process. Except it didn't work until I modified the gdbinit script. So:

  • download https://svn.python.org/projects/python/trunk/Misc/gdbinit and put it in ~/.gdbinit

  • edit it, changing PyEval_EvalFrame to PyEval_EvalFrameEx [edit: no longer needed; the linked file already has this change as of 2010-01-14]

  • Attach gdb: gdb -p PID

  • Get the python stack trace: pystack

Uncommitted answered 6/3, 2009 at 12:49 Comment(3)
The gdbinit at the mentioned URL already seems to have the patch you suggest. In my case, when I typed pystack my CPU just hung. Not sure why.Pyrophotometer
Nope, it doesn't — I was unclear, sorry, because that line appears in three places. The patch I linked to shows which one I had changed when I saw this work.Uncommitted
Like @spiv's answer, this requires the program to run under python compiled with debugging symbols. Otherwise you'll just get No symbol "co" in current context.Greyso
H
19

It can be done with excellent py-spy. It's a sampling profiler for Python programs, so its job is to attach to a Python processes and sample their call stacks. Hence, py-spy dump --pid $SOME_PID is all you need to do to dump call stacks of all threads in the $SOME_PID process. Typically it needs escalated privileges (to read the target process' memory).

Here's an example of how it looks like for a threaded Python application.

$ sudo py-spy dump --pid 31080
Process 31080: python3.7 -m chronologer -e production serve -u www-data -m
Python v3.7.1 (/usr/local/bin/python3.7)

Thread 0x7FEF5E410400 (active): "MainThread"
    _wait (cherrypy/process/wspbus.py:370)
    wait (cherrypy/process/wspbus.py:384)
    block (cherrypy/process/wspbus.py:321)
    start (cherrypy/daemon.py:72)
    serve (chronologer/cli.py:27)
    main (chronologer/cli.py:84)
    <module> (chronologer/__main__.py:5)
    _run_code (runpy.py:85)
    _run_module_as_main (runpy.py:193)
Thread 0x7FEF55636700 (active): "_TimeoutMonitor"
    run (cherrypy/process/plugins.py:518)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)
Thread 0x7FEF54B35700 (active): "HTTPServer Thread-2"
    accept (socket.py:212)
    tick (cherrypy/wsgiserver/__init__.py:2075)
    start (cherrypy/wsgiserver/__init__.py:2021)
    _start_http_thread (cherrypy/process/servers.py:217)
    run (threading.py:865)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)
...
Thread 0x7FEF2BFFF700 (idle): "CP Server Thread-10"
    wait (threading.py:296)
    get (queue.py:170)
    run (cherrypy/wsgiserver/__init__.py:1586)
    _bootstrap_inner (threading.py:917)
    _bootstrap (threading.py:885)  
Harriettharrietta answered 1/4, 2020 at 16:7 Comment(1)
This works, but hypno from @kmaork's answer produces a Python traceback, which is a lot more useful for figuring out where the problem is.Tourcoing
R
14

python -dv yourscript.py

That will make the interpreter to run in debug mode and to give you a trace of what the interpreter is doing.

If you want to interactively debug the code you should run it like this:

python -m pdb yourscript.py

That tells the python interpreter to run your script with the module "pdb" which is the python debugger, if you run it like that the interpreter will be executed in interactive mode, much like GDB

Rap answered 25/9, 2008 at 8:24 Comment(2)
This does not answer the question. The question was about an already running process.Glanders
what is -dv ? Any link to documentation?Diffidence
S
12

I would add this as a comment to haridsv's response, but I lack the reputation to do so:

Some of us are still stuck on a version of Python older than 2.6 (required for Thread.ident), so I got the code working in Python 2.5 (though without the thread name being displayed) as such:

import traceback
import sys
def dumpstacks(signal, frame):
    code = []
    for threadId, stack in sys._current_frames().items():
            code.append("\n# Thread: %d" % (threadId))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    print "\n".join(code)

import signal
signal.signal(signal.SIGQUIT, dumpstacks)
Stambaugh answered 31/3, 2011 at 16:30 Comment(0)
H
12

Take a look at the faulthandler module, new in Python 3.3. A faulthandler backport for use in Python 2 is available on PyPI.

Hussein answered 26/1, 2012 at 13:57 Comment(1)
A more recent answer by @haypo covers this in more detail. I'm not sure how this is usually handled on SO, but it feels wrong to have two essentially duplicate answers...Greyso
S
7

On Solaris, you can use pstack(1) No changes to the python code are necessary. eg.

# pstack 16000 | grep : | head
16000: /usr/bin/python2.6 /usr/lib/pkg.depotd --cfg svc:/application/pkg/serv
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:282 (_wait) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:295 (wait) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/process/wspbus.py:242 (block) ]
[ /usr/lib/python2.6/vendor-packages/cherrypy/_init_.py:249 (quickstart) ]
[ /usr/lib/pkg.depotd:890 (<module>) ]
[ /usr/lib/python2.6/threading.py:256 (wait) ]
[ /usr/lib/python2.6/Queue.py:177 (get) ]
[ /usr/lib/python2.6/vendor-packages/pkg/server/depot.py:2142 (run) ]
[ /usr/lib/python2.6/threading.py:477 (run)
etc.
Stratum answered 28/8, 2011 at 21:30 Comment(2)
There appears to be a Debian/Ubuntu programme pstack that does the same thingSeel
It seems to only give the backtrace under linux, not the Python traceback with filename and line numbers.Prosy
N
7

I was looking for a while for a solution to debug my threads and I found it here thanks to haridsv. I use slightly simplified version employing the traceback.print_stack():

import sys, traceback, signal
import threading
import os

def dumpstacks(signal, frame):
  id2name = dict((th.ident, th.name) for th in threading.enumerate())
  for threadId, stack in sys._current_frames().items():
    print(id2name[threadId])
    traceback.print_stack(f=stack)

signal.signal(signal.SIGQUIT, dumpstacks)

os.killpg(os.getpgid(0), signal.SIGQUIT)

For my needs I also filter threads by name.

Northeastwards answered 15/4, 2012 at 20:28 Comment(0)
C
7

If you're on a Linux system, use the awesomeness of gdb with Python debug extensions (can be in python-dbg or python-debuginfo package). It also helps with multithreaded applications, GUI applications and C modules.

Run your program with:

$ gdb -ex r --args python <programname>.py [arguments]

This instructs gdb to prepare python <programname>.py <arguments> and run it.

Now when you program hangs, switch into gdb console, press Ctr+C and execute:

(gdb) thread apply all py-list

See example session and more info here and here.

Counterreply answered 24/6, 2013 at 8:3 Comment(0)
C
7

You can use the hypno package, like so:

hypno <pid> "import traceback; traceback.print_stack()"

This would print a stack trace into the program's stdout.

Alternatively, if you don't want to print anything to stdout, or you don't have access to it (a daemon for example), you could use the madbg package, which is a python debugger that allows you to attach to a running python program and debug it in your current terminal. It is similar to pyrasite and pyringe, but newer, doesn't require gdb, and uses IPython for the debugger (which means colors and autocomplete).

To see the stack trace of a running program, you could run:

madbg attach <pid>

And in the debugger shell, enter: bt

Disclaimer - I wrote both packages

Claqueur answered 7/8, 2020 at 12:41 Comment(2)
FTW! sudo $(which hypno) $(pgrep pytest) "import traceback; traceback.print_stack()" (It doesn't work without sudo, despite both processes being owned by the same user.)Tourcoing
Great to hear! It can work without sudo if you allow ptrace scope on your machine, more details here: https://mcmap.net/q/25093/-how-to-solve-quot-ptrace-operation-not-permitted-quot-when-trying-to-attach-gdb-to-a-process/2907819Claqueur
R
4

I hacked together some tool which attaches into a running Python process and injects some code to get a Python shell.

See here: https://github.com/albertz/pydbattach

Referendum answered 6/4, 2012 at 18:49 Comment(1)
Note: it's not obvious how to build this. Thanks for the links you've put in README though: pyrasite worked perfectly!Greyso
H
3

It's worth looking at Pydb, "an expanded version of the Python debugger loosely based on the gdb command set". It includes signal managers which can take care of starting the debugger when a specified signal is sent.

A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb.

Hippy answered 29/1, 2009 at 1:28 Comment(1)
Seems it's gone through two (1) rewrites (2) without adding the attach-by-PID feature I was looking for...Greyso
A
2

pyringe is a debugger that can interact with running python processes, print stack traces, variables, etc. without any a priori setup.

While I've often used the signal handler solution in the past, it can still often be difficult to reproduce the issue in certain environments.

Accouter answered 1/5, 2014 at 9:40 Comment(1)
Apparently it's incompatible with certain gdb builds (e.g. the one I had installed on ubuntu): github.com/google/pyringe/issues/16 , requiring rebuilding manually. Another debugger, pyrasite, worked like a charm for me.Greyso
I
2

Since Austin 3.3, you can use the -w/--where option to emit the current stack trace. See https://mcmap.net/q/20373/-can-i-get-the-python-call-stack-with-the-linux-perf

enter image description here

If you want to peek at a running Python application to see the "live" call stack in a top-like fashon you can use austin-tui (https://github.com/p403n1x87/austin-tui). You can install it from PyPI with e.g.

pipx install austin-tui

Note that it requires the austin binary to work (https://github.com/p403n1x87/austin), but then you can attach to a running Python process with

austin-tui -p <pid>
Immixture answered 6/8, 2021 at 19:18 Comment(0)
Y
1

You can use PuDB, a Python debugger with a curses interface to do this. Just add

from pudb import set_interrupt_handler; set_interrupt_handler()

to your code and use Ctrl-C when you want to break. You can continue with c and break again multiple times if you miss it and want to try again.

Yoo answered 27/4, 2013 at 0:54 Comment(1)
When you use the above command in django, don't forget to run a server properly to prevent glitches: "manage.py runserver --noreload --nothreading"Rudimentary
S
1

I am in the GDB camp with the python extensions. Follow https://wiki.python.org/moin/DebuggingWithGdb, which means

  1. dnf install gdb python-debuginfo or sudo apt-get install gdb python2.7-dbg
  2. gdb python <pid of running process>
  3. py-bt

Also consider info threads and thread apply all py-bt.

Sparteine answered 18/4, 2018 at 12:31 Comment(2)
is it normal to get a response like Traceback (most recent call first): Python Exception <class 'gdb.error'> No frame is currently selected.: Error occurred in Python command: No frame is currently selected. when running py-bt in gdb?Hydrophane
nevermind. it's because my app was running as sudo. i needed to run gdb pyton <pid> as sudo, too.Hydrophane
T
1

How to debug any function in console:

Create function where you use pdb.set_trace(), then function you want debug.

>>> import pdb
>>> import my_function

>>> def f():
...     pdb.set_trace()
...     my_function()
... 

Then call created function:

>>> f()
> <stdin>(3)f()
(Pdb) s
--Call--
> <stdin>(1)my_function()
(Pdb) 

Happy debugging :)

Twibill answered 16/1, 2019 at 20:47 Comment(0)
N
0

I don't know of anything similar to java's response to SIGQUIT, so you might have to build it in to your application. Maybe you could make a server in another thread that can get a stacktrace on response to a message of some kind?

Nutritive answered 25/9, 2008 at 9:6 Comment(0)
I
0

There is no way to hook into a running python process and get reasonable results. What I do if processes lock up is hooking strace in and trying to figure out what exactly is happening.

Unfortunately often strace is the observer that "fixes" race conditions so that the output is useless there too.

Impecunious answered 25/9, 2008 at 9:9 Comment(4)
Yeah, this is true. It is a shame though thad pdb does not support attaching to a running process...Impend
This is not true. See the answer by "spiv" above, which shows how to connect gdb and get a Python stack trace.Imtiaz
It's not the same -- those gdb macros aren't reliable and don't provide the full power/familiar interface of pdb. I often wish someone wrote a small app that would use ptrace to inject some Python bytecode into a running Python process and have it execute 'import pdb; pdb.set_trace()', maybe also after temporarily redirecting sys.stdin/stdout.Indecency
This is not true anymore, see other answers pointing to pyringe/pyrasite.Greyso
C
0

use the inspect module.

import inspect help(inspect.stack) Help on function stack in module inspect:

stack(context=1) Return a list of records for the stack above the caller's frame.

I find it very helpful indeed.

Cradling answered 16/9, 2009 at 6:44 Comment(0)
P
0

In Python 3, pdb will automatically install a signal handler the first time you use c(ont(inue)) in the debugger. Pressing Control-C afterwards will drop you right back in there. In Python 2, here's a one-liner which should work even in relatively old versions (tested in 2.7 but I checked Python source back to 2.4 and it looked okay):

import pdb, signal
signal.signal(signal.SIGINT, lambda sig, frame: pdb.Pdb().set_trace(frame))

pdb is worth learning if you spend any amount of time debugging Python. The interface is a bit obtuse but should be familiar to anyone who has used similar tools, such as gdb.

Precocious answered 14/12, 2014 at 2:16 Comment(0)
O
0

In case you need to do this with uWSGI, it has Python Tracebacker built-in and it's just matter of enabling it in the configuration (number is attached to the name for each worker):

py-tracebacker=/var/run/uwsgi/pytrace

Once you have done this, you can print backtrace simply by connecting to the socket:

uwsgi --connect-and-read /var/run/uwsgi/pytrace1
Officious answered 28/4, 2015 at 6:15 Comment(0)
E
-1

At the point where the code is run, you can insert this small snippet to see a nicely formatted printed stack trace. It assumes that you have a folder called logs at your project's root directory.

# DEBUG: START DEBUG -->
import traceback

with open('logs/stack-trace.log', 'w') as file:
    traceback.print_stack(file=file)
# DEBUG: END DEBUG --!
Economic answered 23/5, 2020 at 16:59 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.