How to attach debugger to a python subproccess?

Question

I need to debug a child process spawned by multiprocessing.Process(). The pdb degugger seems to be unaware of forking and unable to attach to already running processes.

Are there any smarter python debuggers which can be attached to a subprocess?

Koph

Answer 1

Winpdb is pretty much the definition of a smarter Python debugger. It explicitly supports going down a fork, not sure it works nicely with multiprocessing.Process() but it's worth a try.

For a list of candidates to check for support of your use case, see the list of Python Debuggers in the wiki.

Cumulostratus

It should be noted that Winpdb is multi-platform, free and Free software. – Marisamariscal

I couldn't, within 20 mins of reading about and playing with winpdb, find a way to just launch an interactive debugging session in an existing script via an import, a la import pdb; pdb.set_trace(). However, the ForkedPdb answer here worked like a charm! – Fragrance

Answer 2

I've been searching for a simple to solution for this problem and came up with this:

import sys
import pdb

class ForkedPdb(pdb.Pdb):
    """A Pdb subclass that may be used
    from a forked multiprocessing child

    """
    def interaction(self, *args, **kwargs):
        _stdin = sys.stdin
        try:
            sys.stdin = open('/dev/stdin')
            pdb.Pdb.interaction(self, *args, **kwargs)
        finally:
            sys.stdin = _stdin

Use it the same way you might use the classic Pdb:

ForkedPdb().set_trace()

Phyte

Nice trick. I will suggest storing sys.stdin when the program starts and using that in the forked process instead of opening /dev/stdin. I don't know the reason but readline works fine then. I will research a bit on this later. – Division

Well, I found a more robust solution keeping the fd of stdin at start. I think readline is always invoked with that one, please read my answer. – Division

mind explaining a little bit how and why does this work? – Clank

Answer 3

This is an elaboration of Romuald's answer which restores the original stdin using its file descriptor. This keeps readline working inside the debugger. Besides, pdb special management of KeyboardInterrupt is disabled, in order it not to interfere with multiprocessing sigint handler.

class ForkablePdb(pdb.Pdb):

    _original_stdin_fd = sys.stdin.fileno()
    _original_stdin = None

    def __init__(self):
        pdb.Pdb.__init__(self, nosigint=True)

    def _cmdloop(self):
        current_stdin = sys.stdin
        try:
            if not self._original_stdin:
                self._original_stdin = os.fdopen(self._original_stdin_fd)
            sys.stdin = self._original_stdin
            self.cmdloop()
        finally:
            sys.stdin = current_stdin

Division

This keeps readline working inside the debugger what does this mean ? – Prussian

It restores the stream readline is attached to as the stdin of the current process. – Division

how is that different from above answer...I mean what difference does it make or what more it adds...sry for being naive....have less info on this – Prussian

It works when you are debugging multiple processes, besides the sigint feature, as far as I can remember. – Division

so as per your solution, pdb will work even if we span a new process through multiprocess ? – Prussian

That was the idea. – Division

@memeplex: I tried your class, and I am getting: ` File "multi_proc.py", line 28, in _cmdloop self._original_stdin = os.fdopen(self._original_stdin_fd) File "/usr/lib/python3.6/os.py", line 1017, in fdopen return io.open(fd, *args, **kwargs) OSError: [Errno 9] Bad file descriptor` – Awful

This one allows you to use the up arrow to recall previously-used commands, a nice improvement – Pernicious

@Division getting OSError: [Errno 9] Bad file descriptor error (python 3) – Whipstall

Answer 4

remote-pdb can be used to debug sub-processes. After installation, put the following lines in the code you need to debug:

import remote_pdb
remote_pdb.set_trace()

remote-pdb will print a port number which will accept a telnet connection for debugging that specific process. There are some caveats around worker launch order, where stdout goes when using various frontends, etc. To ensure a specific port is used (must be free and accessible to the current user), use the following instead:

from remote_pdb import RemotePdb
RemotePdb('127.0.0.1', 4444).set_trace()

remote-pdb may also be launched via the breakpoint() command in Python 3.7.

Octahedrite

Answer 5

Building upon @memplex idea, I had to modify it to get it to work with joblib by setting the sys.stdin in the constructor as well as passing it directly along via joblib.

import os
import pdb
import signal
import sys
import joblib

_original_stdin_fd = None

class ForkablePdb(pdb.Pdb):
    _original_stdin = None
    _original_pid = os.getpid()

    def __init__(self):
        pdb.Pdb.__init__(self)
        if self._original_pid != os.getpid():
            if _original_stdin_fd is None:
                raise Exception("Must set ForkablePdb._original_stdin_fd to stdin fileno")

            self.current_stdin = sys.stdin
            if not self._original_stdin:
                self._original_stdin = os.fdopen(_original_stdin_fd)
            sys.stdin = self._original_stdin

    def _cmdloop(self):
        try:
            self.cmdloop()
        finally:
            sys.stdin = self.current_stdin

def handle_pdb(sig, frame):
    ForkablePdb().set_trace(frame)

def test(i, fileno):
    global _original_stdin_fd
    _original_stdin_fd = fileno
    while True:
        pass    

if __name__ == '__main__':
    print "PID: %d" % os.getpid()
    signal.signal(signal.SIGUSR2, handle_pdb)
    ForkablePdb().set_trace()
    fileno = sys.stdin.fileno()
    joblib.Parallel(n_jobs=2)(joblib.delayed(test)(i, fileno) for i in range(10))

Gaylordgaylussac

Answer 6

Just use PuDB that gives you an awesome TUI (GUI on terminal) and supports multiprocessing as follow:

from pudb import forked; forked.set_trace()

Boat

Answer 7

The problem here is that Python always connects sys.stdin in the child process to os.devnull to avoid contention for the stream. But this means that when the debugger (or a simple input()) tries to connect to stdin to get input from the user, it immediately reaches end-of-file and reports an error.

One solution, at least if you don't expect multiple debuggers to run at the same time, is to reopen stdin in the child process. That can be done by setting sys.stdin to open(0), which always opens the active terminal. This in fact is what the ForkedPdb solution does, but it can be done more simply and in an os-independent manner like this:

import multiprocessing, sys

def main():
    process = multiprocessing.Process(target=worker)
    process.start()
    process.join()

def worker():
    # Python automatically closes sys.stdin for the subprocess, so we reopen
    # stdin. This enables pdb to connect to the terminal and accept commands.
    # See https://mcmap.net/q/169700/-python-multiprocessing-stdin-input.
    sys.stdin = open(0) # or os.fdopen(0)
    print("Hello from the subprocess.")
    breakpoint()  # or import pdb; pdb.set_trace()
    print("Exited from breakpoint in the subprocess.")

if __name__ == '__main__':
    main()

Ann

the open(0) needs to happen first thing in the worker, as in this example. My first go at this did it further in the body of worker, and the open threw a bad filedescriptor error. Moving it to the top fixed that. – Karlee

Answer 8

An idea I had was to create "dummy" classes to fake the implementation of the methods you are using from multiprocessing:

from multiprocessing import Pool


class DummyPool():
    @staticmethod
    def apply_async(func, args, kwds):
        return DummyApplyResult(func(*args, **kwds))

    def close(self): pass
    def join(self): pass


class DummyApplyResult():
    def __init__(self, result):
        self.result = result

    def get(self):
        return self.result


def foo(a, b, switch):
    # set trace when DummyPool is used
    # import ipdb; ipdb.set_trace()
    if switch:
        return b - a
    else:
        return a - b


if __name__ == '__main__':
    xml = etree.parse('C:/Users/anmendoza/Downloads/jim - 8.1/running-config.xml')
    pool = DummyPool()  # switch between Pool() and DummyPool() here
    results = []
    results.append(pool.apply_async(foo, args=(1, 100), kwds={'switch': True}))
    pool.close()
    pool.join()
    results[0].get()

Hinkle

Answer 9

Here is the version of the ForkedPdb(Romuald's Solution) which will work for Windows and *nix based systems.

import sys
import pdb
import win32console


class MyHandle():
    def __init__(self):
        self.screenBuffer = win32console.GetStdHandle(win32console.STD_INPUT_HANDLE)
    
    def readline(self):
        return self.screenBuffer.ReadConsole(1000)

class ForkedPdb(pdb.Pdb):
    def interaction(self, *args, **kwargs):
        _stdin = sys.stdin
        try:
            if sys.platform == "win32":
                sys.stdin = MyHandle()
            else:
                sys.stdin = open('/dev/stdin')
            pdb.Pdb.interaction(self, *args, **kwargs)
        finally:
            sys.stdin = _stdin

Collectanea

Answer 10

If you are on a supported platform, try DTrace. Most of the BSD / Solaris / OS X family support DTrace.

Here is an intro by the author. You can use Dtrace to debug just about anything.

Here is a SO post on learning DTrace.

Nates