How to make pdb recognize that the source has changed between runs?
Asked Answered
I

7

22

From what I can tell, pdb does not recognize when the source code has changed between "runs". That is, if I'm debugging, notice a bug, fix that bug, and rerun the program in pdb (i.e. without exiting pdb), pdb will not recompile the code. I'll still be debugging the old version of the code, even if pdb lists the new source code.

So, does pdb not update the compiled code as the source changes? If not, is there a way to make it do so? I'd like to be able to stay in a single pdb session in order to keep my breakpoints and such.

FWIW, gdb will notice when the program it's debugging changes underneath it, though only on a restart of that program. This is the behavior I'm trying to replicate in pdb.

Iridescence answered 7/4, 2009 at 10:9 Comment(0)
K
4

What do you mean by "rerun the program in pdb?" If you've imported a module, Python won't reread it unless you explicitly ask to do so, i.e. with reload(module). However, reload is far from bulletproof (see xreload for another strategy).

There are plenty of pitfalls in Python code reloading. To more robustly solve your problem, you could wrap pdb with a class that records your breakpoint info to a file on disk, for example, and plays them back on command.

(Sorry, ignore the first version of this answer; it's early and I didn't read your question carefully enough.)

Krishna answered 7/4, 2009 at 11:44 Comment(5)
What I mean by "rerun" is, conceptually, what I mean by rerunning a python program from the command line. pdb clearly understands the notion of my program exiting, so I'm wondering, when it runs my program a second time, if I can get it to also recompile the source as necessary.Iridescence
You can try using reload from inside pdb before rerunning it, but again, depending on structure of your program, it may not be reliable. (FWIW, I consider this the single biggest failure of Python as a language. Coming from environments like Smalltalk and Lisp, it's just depressing.)Krishna
@NicholasRiley I would like to incorporate xreload as a command in the python trepan debuggers (trepan3k pypi.python.org/pypi/trepan3k and trepan2 pypi.python.org/pypi/trepan2). These are GPL3. Is this okay? Are you the author?Linkboy
@NicholasRiley, can you given an example of To more robustly solve your problem, you could wrap pdb with a class that records your breakpoint info to a file on disk, for example, and plays them back on command.? I face this problem pretty regularly while debugging and am trying to see if there is a robust way of reloading modules..Loyal
Honestly it's been 9 years since I wrote the above and there are multiple decent GUI Python debuggers I'd use now instead rather than trying to hack pdb to persist breakpoints (which would break in any case if you edited your file and the line numbers changed). If you really want to persist breakpoints in pdb, see https://mcmap.net/q/589342/-save-breakpoints-to-file for an example.Krishna
T
7

The following mini-module may help. If you import it in your pdb session, then you can use:

pdb> pdbs.r()

at any time to force-reload all non-system modules except main. The code skips that because it throws an ImportError('Cannot re-init internal module main') exception.

# pdbs.py - PDB support

from __future__ import print_function

def r():
    """Reload all non-system modules, to reload stuff on pbd restart. """
    import importlib
    import sys

    # This is likely to be OS-specific
    SYS_PREFIX = '/usr/lib'

    for k, v in list(sys.modules.items()):
        if (
            k == "__main__" or
            k.startswith("pdb") or
            not getattr(v, "__file__", None)
            or v.__file__.startswith(SYS_PREFIX)
        ):
            continue
        print("reloading %s [%s]" % (k, v.__file__), file=sys.stderr)
        importlib.reload(v)
Tessin answered 21/4, 2014 at 23:22 Comment(2)
When I run pdb.r(), I get: *** AttributeError: 'dict' object has no attribute 'iteritems'Grapnel
My apologies - .iteritems() is not compatible with Python 3.x. I have modified the code so that it should work.Tessin
P
5

Based on @pourhaus answer (from 2014), this recipe augments the pdb++ debugger with a reload command (expected to work on both Linux & Windows, on any Python installation).

TIP: the new reload command accepts an optional list of module-prefixes to reload (and to exclude), not to break already loaded globals when resuming debugging.

Just insert the following Python-3.6 code into your ~/.pdbrc.py file:

## Augment `pdb++` with a `reload` command
#
#  See https://mcmap.net/q/580705/-how-to-make-pdb-recognize-that-the-source-has-changed-between-runs/64194585#64194585

from pdb import Pdb


def _pdb_reload(pdb, modules):
    """
    Reload all non system/__main__ modules, without restarting debugger.

    SYNTAX:
        reload [<reload-module>, ...] [-x [<exclude-module>, ...]]

    * a dot(`.`) matches current frame's module `__name__`;
    * given modules are matched by prefix;
    * any <exclude-modules> are applied over any <reload-modules>.

    EXAMPLES:
        (Pdb++) reload                  # reload everything (brittle!)
        (Pdb++) reload  myapp.utils     # reload just `myapp.utils`
        (Pdb++) reload  myapp  -x .     # reload `myapp` BUT current module

    """
    import importlib
    import sys

    ## Derive sys-lib path prefix.
    #
    SYS_PREFIX = importlib.__file__
    SYS_PREFIX = SYS_PREFIX[: SYS_PREFIX.index("importlib")]

    ## Parse args to decide prefixes to Include/Exclude.
    #
    has_excludes = False
    to_include = set()
    # Default prefixes to Exclude, or `pdb++` will break.
    to_exclude = {"__main__", "pdb", "fancycompleter", "pygments", "pyrepl"}
    for m in modules.split():
        if m == "-x":
            has_excludes = True
            continue

        if m == ".":
            m = pdb._getval("__name__")

        if has_excludes:
            to_exclude.add(m)
        else:
            to_include.add(m)

    to_reload = [
        (k, v)
        for k, v in sys.modules.items()
        if (not to_include or any(k.startswith(i) for i in to_include))
        and not any(k.startswith(i) for i in to_exclude)
        and getattr(v, "__file__", None)
        and not v.__file__.startswith(SYS_PREFIX)
    ]
    print(
        f"PDB-reloading {len(to_reload)} modules:",
        *[f"  +--{k:28s}:{getattr(v, '__file__', '')}" for k, v in to_reload],
        sep="\n",
        file=sys.stderr,
    )

    for k, v in to_reload:
        try:
            importlib.reload(v)
        except Exception as ex:
            print(
                f"Failed to PDB-reload module: {k} ({v.__file__}) due to: {ex!r}",
                file=sys.stderr,
            )


Pdb.do_reload = _pdb_reload
Pliable answered 4/10, 2020 at 12:27 Comment(1)
Got error TypeError: super(type, obj): obj must be an instance or subtype of type. I seem to encounter this error before in IPython.Agha
K
4

What do you mean by "rerun the program in pdb?" If you've imported a module, Python won't reread it unless you explicitly ask to do so, i.e. with reload(module). However, reload is far from bulletproof (see xreload for another strategy).

There are plenty of pitfalls in Python code reloading. To more robustly solve your problem, you could wrap pdb with a class that records your breakpoint info to a file on disk, for example, and plays them back on command.

(Sorry, ignore the first version of this answer; it's early and I didn't read your question carefully enough.)

Krishna answered 7/4, 2009 at 11:44 Comment(5)
What I mean by "rerun" is, conceptually, what I mean by rerunning a python program from the command line. pdb clearly understands the notion of my program exiting, so I'm wondering, when it runs my program a second time, if I can get it to also recompile the source as necessary.Iridescence
You can try using reload from inside pdb before rerunning it, but again, depending on structure of your program, it may not be reliable. (FWIW, I consider this the single biggest failure of Python as a language. Coming from environments like Smalltalk and Lisp, it's just depressing.)Krishna
@NicholasRiley I would like to incorporate xreload as a command in the python trepan debuggers (trepan3k pypi.python.org/pypi/trepan3k and trepan2 pypi.python.org/pypi/trepan2). These are GPL3. Is this okay? Are you the author?Linkboy
@NicholasRiley, can you given an example of To more robustly solve your problem, you could wrap pdb with a class that records your breakpoint info to a file on disk, for example, and plays them back on command.? I face this problem pretty regularly while debugging and am trying to see if there is a robust way of reloading modules..Loyal
Honestly it's been 9 years since I wrote the above and there are multiple decent GUI Python debuggers I'd use now instead rather than trying to hack pdb to persist breakpoints (which would break in any case if you edited your file and the line numbers changed). If you really want to persist breakpoints in pdb, see https://mcmap.net/q/589342/-save-breakpoints-to-file for an example.Krishna
C
2

I decided to comment some lines in my input script, and after

(Pdb) run

I got pdb to recognize that change. The bad thing: it runs the script from the beginning. The good things below.

(Pdb) help run
run [args...]
        Restart the debugged python program. If a string is supplied
        it is split with "shlex", and the result is used as the new
        sys.argv.  History, breakpoints, actions and debugger options
        are preserved.  "restart" is an alias for "run".
Cortex answered 30/10, 2019 at 14:26 Comment(1)
Surprisingly this works. It's a tad counterintuitive from a compiled language background...Groundsel
A
1

I'm new to Python, but I had a thought about the question as it exists now, after reading the other answers. If you "notice a bug, fix that bug, and rerun the program in pdb (i.e. without exiting pdb)," then it is likely that the state you'd like pdb to keep using (breakpoints, commands, etc.) won't serve as well, specifically because line numbers may have changed, and any logic related to breakpoints and the conditions they use may be broken.

For useful debugging commands that you don't want to type repeatedly, the docs say "If a file .pdbrc exists in the user’s home directory or in the current directory, it is read with 'utf-8' encoding and executed as if it had been typed at the debugger prompt."

The last big problem to solve here is how to preserve a useful commands command in pdb, since alias doesn't specify how to alias multi-line pdb commands (of which commands is the only one I know).

Azriel answered 9/8, 2023 at 16:46 Comment(0)
U
0

May not work for more complex programs, but for a simple example using importlib.reload() using Python v3.5.3:

[user@machine ~] cat test.py
print('Test Message')

#
# start and run with debugger
#
[user@machine ~] python3 -m pdb test.py
> /home/user/test.py(1)<module>()
-> print('Test Message')
(Pdb) c
Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Test Message')

#
# in another terminal, change test.py to say "Changed Test Message"
#

#
# back in PDB:
#
(Pdb) import importlib; import test; importlib.reload(test)
Changed Test Message
<module 'test' from '/home/user/test.py'>
(Pdb) c
Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Changed Test Message')
(Pdb) c
Changed Test Message
The program finished and will be restarted
> /home/user/test.py(1)<module>()
-> print('Changed Test Message')
Undershorts answered 7/10, 2021 at 13:2 Comment(0)
N
-1

ipdb %autoreload extension

6.2.0 docs document http://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html#module-IPython.extensions.autoreload :

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: from foo import some_function

In [4]: some_function()
Out[4]: 42

In [5]: # open foo.py in an editor and change some_function to return 43

In [6]: some_function()
Out[6]: 43
Nicholasnichole answered 21/11, 2017 at 14:3 Comment(1)
Yeah.. this is not a correct answer. The %autoreload extension doesn't actually reload your module if you already in the debugger. The reloading only works if you are working in ipython. if you re-execute the code, after making changes to the code in ipython, then it will reload the code. Otherwise not.Loyal

© 2022 - 2024 — McMap. All rights reserved.