How to debug the stack trace that causes a subsequent exception in python?

Question

Python (and ipython) has very powerful post-mortem debugging capabilities, allowing variable inspection and command execution at each scope in the traceback. The up/down debugger commands allow changing frame for the stack trace of the final exception, but what about the __cause__ of that exception, as defined by the raise ... from ... syntax?

Python 3.7.6 (default, Jan  8 2020, 13:42:34) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: def foo(): 
   ...:     bab = 42 
   ...:     raise TypeError 
   ...:                                                                                                                                      

In [2]: try: 
   ...:     foo() 
   ...: except TypeError as err: 
   ...:     barz = 5 
   ...:     raise ValueError from err 
   ...:                                                                                                                                      
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-dd046d7cece0> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:

<ipython-input-1-da9a05838c59> in foo()
      2     bab = 42
----> 3     raise TypeError
      4 

TypeError: 

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-2-dd046d7cece0> in <module>
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError from err
      6 

ValueError: 

In [3]: %debug                                                                                                                               
> <ipython-input-2-dd046d7cece0>(5)<module>()
      2     foo()
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError from err
      6 

ipdb> barz                                                                                                                                   
5
ipdb> bab                                                                                                                                    
*** NameError: name 'bab' is not defined
ipdb> down                                                                                                                                   
*** Newest frame
ipdb> up                                                                                                                                     
*** Oldest frame

Is there a way to access bab from the debugger?

EDIT: I realized post-mortem debugging isn't just a feature of ipython and ipdb, it's actually part of vanilla pdb. The above can also be reproduced by putting the code into a script testerr.py and running python -m pdb testerr.py and running continue. After the error, it says

Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program

and gives a debugger at the same spot.

Quechuan

It might make sense to file a feature request for IPython postmortem debugging to be able to follow __cause__ and __context__ chains. – Aintab

Apparently post-mortem debugging is a vanilla pdb feature, would it make sense for pdb to support following __cause__ and __context__ chains? – Quechuan

Answer 1

You can use the with_traceback(tb) method to preserve the original exception's traceback:

try: 
    foo()
except TypeError as err:
    barz = 5
    raise ValueError().with_traceback(err.__traceback__) from err

Note that I have updated the code to raise an exception instance rather than the exception class.

Here is the full code snippet in iPython:

In [1]: def foo(): 
   ...:     bab = 42 
   ...:     raise TypeError() 
   ...:                                                                                                                                                         

In [2]: try: 
   ...:     foo() 
   ...: except TypeError as err: 
   ...:     barz = 5 
   ...:     raise ValueError().with_traceback(err.__traceback__) from err 
   ...:                                                                                                                                                         
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-a5a6d81e4c1a> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:

<ipython-input-1-ca1efd1bee60> in foo()
      2     bab = 42
----> 3     raise TypeError()
      4 

TypeError: 

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-2-a5a6d81e4c1a> in <module>
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError().with_traceback(err.__traceback__) from err
      6 

<ipython-input-2-a5a6d81e4c1a> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:
      4     barz = 5
      5     raise ValueError().with_traceback(err.__traceback__) from err

<ipython-input-1-ca1efd1bee60> in foo()
      1 def foo():
      2     bab = 42
----> 3     raise TypeError()
      4 

ValueError: 

In [3]: %debug                                                                                                                                                  
> <ipython-input-1-ca1efd1bee60>(3)foo()
      1 def foo():
      2     bab = 42
----> 3     raise TypeError()
      4 

ipdb> bab                                                                                                                                                       
42
ipdb> u                                                                                                                                                         
> <ipython-input-2-a5a6d81e4c1a>(2)<module>()
      1 try:
----> 2     foo()
      3 except TypeError as err:
      4     barz = 5
      5     raise ValueError().with_traceback(err.__traceback__) from err

ipdb> u                                                                                                                                                         
> <ipython-input-2-a5a6d81e4c1a>(5)<module>()
      2     foo()
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError().with_traceback(err.__traceback__) from err
      6 

ipdb> barz                                                                                                                                                      
5

EDIT - An alternative inferior approach

Addressing @user2357112supportsMonica's first comment, if you wish to avoid multiple dumps of the original exception's traceback in the log, it's possible to raise from None. However, as @user2357112supportsMonica's second comment states, this hides the original exception's message. This is particularly problematic in the common case where you're not post-mortem debugging but rather inspecting a printed traceback.

try: 
    foo()
except TypeError as err:
    barz = 5
    raise ValueError().with_traceback(err.__traceback__) from None

Here is the code snippet in iPython:

In [4]: try: 
   ...:     foo() 
   ...: except TypeError as err: 
   ...:     barz = 5 
   ...:     raise ValueError().with_traceback(err.__traceback__) from None    
   ...:                                                                                                                                                         
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-b090fb9c510e> in <module>
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError().with_traceback(err.__traceback__) from None
      6 

<ipython-input-6-b090fb9c510e> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:
      4     barz = 5
      5     raise ValueError().with_traceback(err.__traceback__) from None

<ipython-input-2-ca1efd1bee60> in foo()
      1 def foo():
      2     bab = 42
----> 3     raise TypeError()
      4 

ValueError: 

In [5]: %debug                                                                                                                                                  
> <ipython-input-2-ca1efd1bee60>(3)foo()
      1 def foo():
      2     bab = 42
----> 3     raise TypeError()
      4 

ipdb> bab                                                                                                                                                       
42
ipdb> u                                                                                                                                                         
> <ipython-input-6-b090fb9c510e>(2)<module>()
      1 try:
----> 2     foo()
      3 except TypeError as err:
      4     barz = 5
      5     raise ValueError().with_traceback(err.__traceback__) from None

ipdb> u                                                                                                                                                         
> <ipython-input-6-b090fb9c510e>(5)<module>()
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError().with_traceback(err.__traceback__) from None
      6 

ipdb> barz                                                                                                                                                      
5

Raising from None is required since otherwise the chaining would be done implicitly, attaching the original exception as the new exception’s __context__ attribute. Note that this differs from the __cause__ attribute which is set when the chaining is done explicitly.

In [6]: try: 
   ...:     foo() 
   ...: except TypeError as err: 
   ...:     barz = 5 
   ...:     raise ValueError().with_traceback(err.__traceback__) 
   ...:                                                                                                                                                         
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-ee78991171cb> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:

<ipython-input-2-ca1efd1bee60> in foo()
      2     bab = 42
----> 3     raise TypeError()
      4 

TypeError: 

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-5-ee78991171cb> in <module>
      3 except TypeError as err:
      4     barz = 5
----> 5     raise ValueError().with_traceback(err.__traceback__)
      6 

<ipython-input-5-ee78991171cb> in <module>
      1 try:
----> 2     foo()
      3 except TypeError as err:
      4     barz = 5
      5     raise ValueError().with_traceback(err.__traceback__)

<ipython-input-2-ca1efd1bee60> in foo()
      1 def foo():
      2     bab = 42
----> 3     raise TypeError()
      4 

ValueError: 

Onrush

The problem here is that exception chaining is supposed to be the way to preserve information about the original exception, including its traceback. You can see in the output that the stack trace includes two copies of the original exception's stack trace. This can make it hard to see where the second exception actually occurred. – Aintab

@user2357112supportsMonica, thanks for your input. I suggested an alternative implementation that addresses your concern. – Onrush

But now the traceback won't show the original exception's message. (The original exception is still accessible as the new exception's __context__, but this is obscure and hard to discover, and it doesn't help the common case where you're not post-mortem debugging and all you have is a printed stack trace.) – Aintab

@user2357112supportsMonica, that's an important observation which I originally neglected to include in my answer. Thank you for the feedback! – Onrush

I'm also a little surprised that the with_traceback method appends the traceback instead of prepending it, since it means that "(most recent call last)" is no longer the case. – Quechuan

Answer 2

Yoel answer works and should be your go-to procedure, but if the trace is a bit harder to debug, you may instead use the trace module.

The trace module will print out each instruction executed, line by line. There is a catch, though. Standard library and package calls will also be traced, and this likely means that the trace will be flooded with code that is not meaningful.

To avoid this behavior, you may pass the --ignore-dir argument with the location of your Python library and site packages folder.

Run python -m site to find the locations of your site packages, then call trace with the following arguments:

python -m trace --trace --ignore-dir=/usr/lib/python3.8:/usr/local/lib/python3.8/dist-packages main.py args

Replacing the ignore-dir with all folders and the main.py args with a script location and arguments.

You may also use the Trace module directly in your code if you want to run a certain function, refer to this example extracted from https://docs.python.org/3.0/library/trace.html:

import sys
import trace

# create a Trace object, telling it what to ignore, and whether to
# do tracing or line-counting or both.
tracer = trace.Trace(
    ignoredirs=[sys.prefix, sys.exec_prefix],
    trace=0,
    count=1)

# run the new command using the given tracer
tracer.run('main()')

# make a report, placing output in /tmp
r = tracer.results()
r.write_results(show_missing=True, coverdir="/tmp")

Gainly

That trace command is really cool! It seems pretty broadly applicable. Using the Trace instance directly works for running, but the write_results method seems to silently fail - I can't get it to actually output anything to file. – Quechuan

I've never used the write_results method so I can't tell if it works. I've been using the trace command directly rather than hardcoding the class in my code. You can always log the output to a file using tee. – Gainly

Answer 3

I also just found a way to do this without modifying the underlying source code - simply running commands in the post-mortem debugger.

I saw from this answer you can get the locals directly from the traceback instance.

(Pdb) ll
  1  -> def foo():
  2         bab = 42
  3         raise TypeError
  4   
  5     try:
  6         foo()
  7     except TypeError as err:
  8         barz = 5
  9  >>     raise ValueError from err
 10
(Pdb) err # not sure why err is not defined
*** NameError: name 'err' is not defined
(Pdb) import sys
(Pdb) sys.exc_info()
(<class 'AttributeError'>, AttributeError("'Pdb' object has no attribute 'do_sys'"), <traceback object at 0x107cb5be0>)
(Pdb) err = sys.exc_info()[1].__context__
(Pdb) err # here we go
ValueError()
(Pdb) err.__cause__
TypeError()
(Pdb) err.__traceback__.tb_next.tb_next.tb_next.tb_next.tb_frame.f_locals['barz']
5
(Pdb) err.__cause__.__traceback__.tb_next.tb_frame.f_locals['bab']
42

Quechuan

err isn't defined in the post-mortem debugger because it's cleared at the end of the except clause in order to avoid a reference cycle with the stack frame ("exception -> traceback -> stack frame -> exception"), which would have kept all locals in that frame alive until the next garbage collection occurs. – Onrush

Answer 4

Because sys.last_value.__traceback__ is sys.last_traceback, and ipdb makes use of the latter, you can simply move along a chain of Exceptions, and debug at the desired level by overwriting it. Starting at sys.last_value, walk up the val.__context__ chain to the desired level (new), then set sys.last_traceback = new.__traceback__, and invoke %debug.

I wrote a small IPython magic, %chain, to make it easy to inspect the exception chain, move to an arbitrary or relative depth, or to the end of the chain, and debug. Just drop it in your iPython startup directory (e.g. ~/.ipython/profile_default/startup, and %chain -h for usage.)

Pontificate

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Klockau

Added some context. – Pontificate