How do I dump an entire Python process for later debugging inspection?

Question

I have a Python application in a strange state. I don't want to do live debugging of the process. Can I dump it to a file and examine its state later? I know I've restored corefiles of C programs in gdb later, but I don't know how to examine a Python application in a useful way from gdb.

(This is a variation on my question about debugging memleaks in a production system.)

Lamonica

Note also that a QUIT signal (Ctrl-\) dumps core too. – Florindaflorine

Related: #6132969 #14519600 – Klos

to print the stack trace of a running Python process, or of a Python core dump, pystack could be used github.com/bloomberg/pystack#pystack – Halda

Answer 1

There is no builtin way other than aborting (with os.abort(), causing the coredump if resource limits allow it) -- although you can certainly build your own 'dump' function that dumps relevant information about the data you care about. There are no ready-made tools for it.

As for handling the corefile of a Python process, the Python source has a gdbinit file that contains useful macros. It's still a lot more painful than somehow getting into the process itself (with pdb or the interactive interpreter) but it makes life a little easier.

Amazonas

You could add a signal handler to dump the Python state and open that in a debugger using github.com/gooli/pydump. – Samella

Since Thomas Wouters' original response, it became possible to extend gdb with Python itself, and David Malcolm wrote an awesome extension that provides much more functionality than the gdbinit file. The great thing is that it all comes with many Linux distros now (like Ubuntu 14.04) and doesn't require any additional compilation. I've documented my own explorations with it in Adventures in Python Core Dumping for anyone who's interested. – Braxy

Answer 2

If you only care about storing the traceback object (which is all you need to start a debugging session), you can use debuglater (a fork of pydump). It works with recent versions of Python and has a IPython/Jupyter integration.

If you want to store the entire session, look at dill. It has a dump_session, and load_session functions.

Here are two other relevant projects:

• python-checkpointing2
• pycrunch-trace

If you're looking for a language agnostic solution, you want to create a core dump file. Here's an example with Python.

Liquesce

Answer 3

Someone above said that there is no builtin way to perform this, but that's not entirely true. For an example, you could take a look at the pylons debugging tools. Whene there is an exception, the exception handler saves the stack trace and prints a URL on the console that can be used to retrieve the debugging session over HTTP.

While they're probably keeping these sessions in memory, they're just python objects, so there's nothing to stop you from pickling a stack dump and restoring it later for inspection. It would mean some changes to the app, but it should be possible...

After some research, it turns out the relevant code is actually coming from Paste's EvalException module. You should be able to look there to figure out what you need.

Linin

Stack traces are readily obtainable, but I'm more interested in the entire set of allocated objects than I am in what happens to be in the stack at the moment. – Lamonica

Answer 4

It's also possible to write something that would dump all the data from the process, e.g.

• Pickler that ignores the objects it can't pickle (replacing them with something else) (e.g. Python: Pickling a dict with some unpicklable items)
• Method that recursively converts everything into serializable stuff (e.g. this, except it needs a check for infinitely recursing objects and do something with those; also it could try dir() and getattr() to process some of the unknown objects, e.g. extension classes).

But leaving a running process with manhole or pylons or something like that certainly seems more convenient when possible.

(also, I wonder if something more convenient was written since this question was first asked)

Berkie

Answer 5

This answer suggests making your program core dump and then continuing execution on another sufficiently similar box.

Igloo