How to limit python traceback to specific files
Asked Answered
C

3

46

I write a lot of Python code that uses external libraries. Frequently I will write a bug, and when I run the code I get a big long traceback in the Python console. 99.999999% of the time it's due to a coding error in my code, not because of a bug in the package. But the traceback goes all the way to the line of error in the package code, and either it takes a lot of scrolling through the traceback to find the code I wrote, or the traceback is so deep into the package that my own code doesn't even appear in the traceback.

Is there a way to "black-box" the package code, or somehow only show traceback lines from my code? I'd like the ability to specify to the system which directories or files I want to see traceback from.

Continuate answered 11/8, 2015 at 18:52 Comment(3)
You could catch the exception in main or lower, and decide what to print. Use the traceback module. extract_tb() can be used to retrieve the values. You can look at them to decide what to print, and what not to.Ac
As an alternative that wouldn't require wrapping all your existing code in try/except blocks, you could do what cgitb does and use sys.excepthook to get the desired results. There might even be a way to configure cgitb to do it, but I'm not sure. (Typically it's for expanding traceback output.)Osteoid
@Two-BitAlchemist, the sys.excepthook method looks great in general, but I'm trying to debug a Flask application. I can't figure out, even in debug mode where it's own error handling system is bypassed, how to overwrite the default sys.excepthook.Continuate
I
24

In order to print your own stacktrace, you would need to handle all unhandled exceptions yourself; this is how the sys.excepthook becomes handy.

The signature for this function is sys.excepthook(type, value, traceback) and its job is:

This function prints out a given traceback and exception to sys.stderr.

So as long as you can play with the traceback and only extract the portion you care about you should be fine. Testing frameworks do that very frequently; they have custom assert functions which usually does not appear in the traceback, in other words they skip the frames that belong to the test framework. Also, in those cases, the tests usually are started by the test framework as well.

You end up with a traceback that looks like this:

[ custom assert code ] + ... [ code under test ] ... + [ test runner code ]

How to identify your code.

You can add a global to your code:

__mycode = True

Then to identify the frames:

def is_mycode(tb):
  globals = tb.tb_frame.f_globals
  return globals.has_key('__mycode')

How to extract your frames.

  1. skip the frames that don't matter to you (e.g. custom assert code)
  2. identify how many frames are part of your code -> length
  3. extract length frames

    def mycode_traceback_levels(tb):
      length = 0
      while tb and is_mycode(tb):
        tb = tb.tb_next
        length += 1
      return length
    

Example handler.

def handle_exception(type, value, tb):
  # 1. skip custom assert code, e.g.
  # while tb and is_custom_assert_code(tb):
  #   tb = tb.tb_next
  # 2. only display your code
  length = mycode_traceback_levels(tb)
  print ''.join(traceback.format_exception(type, value, tb, length))

install the handler:

sys.excepthook = handle_exception

What next?

You could adjust length to add one or more levels if you still want some info about where the failure is outside of your own code.

see also https://gist.github.com/dnozay/b599a96dc2d8c69b84c6

Imtiaz answered 7/10, 2015 at 18:13 Comment(0)
W
9

As others suggested, you could use sys.excepthook:

This function prints out a given traceback and exception to sys.stderr.

When an exception is raised and uncaught, the interpreter calls sys.excepthook with three arguments, the exception class, exception instance, and a traceback object. In an interactive session this happens just before control is returned to the prompt; in a Python program this happens just before the program exits. The handling of such top-level exceptions can be customized by assigning another three-argument function to sys.excepthook.

(emphasis mine)

It's possible to filter a traceback extracted by extract_tb (or similar functions from the traceback module) based on specified directories.

Two functions that can help:

from os.path import join, abspath
from traceback import extract_tb, format_list, format_exception_only

def spotlight(*show):
    ''' Return a function to be set as new sys.excepthook.
        It will SHOW traceback entries for files from these directories. '''
    show = tuple(join(abspath(p), '') for p in show)

    def _check_file(name):
        return name and name.startswith(show)

    def _print(type, value, tb):
        show = (fs for fs in extract_tb(tb) if _check_file(fs.filename))
        fmt = format_list(show) + format_exception_only(type, value)
        print(''.join(fmt), end='', file=sys.stderr)

    return _print

def shadow(*hide):
    ''' Return a function to be set as new sys.excepthook.
        It will HIDE traceback entries for files from these directories. '''
    hide = tuple(join(abspath(p), '') for p in hide)

    def _check_file(name):
        return name and not name.startswith(hide)

    def _print(type, value, tb):
        show = (fs for fs in extract_tb(tb) if _check_file(fs.filename))
        fmt = format_list(show) + format_exception_only(type, value)
        print(''.join(fmt), end='', file=sys.stderr)

    return _print

They both use the traceback.extract_tb. It returns "a list of “pre-processed” stack trace entries extracted from the traceback object"; all of them are instances of traceback.FrameSummary (a named tuple). Each traceback.FrameSummary object has a filename field which stores the absolute path of the corresponding file. We check if it starts with any of the directory paths provided as separate function arguments to determine if we'll need to exclude the entry (or keep it).


Here's an Example:

The enum module from the standard library doesn't allow reusing keys,

import enum
enum.Enum('Faulty', 'a a', module=__name__)

yields

Traceback (most recent call last):
  File "/home/vaultah/so/shadows/main.py", line 23, in <module>
    enum.Enum('Faulty', 'a a', module=__name__)
  File "/home/vaultah/cpython/Lib/enum.py", line 243, in __call__
    return cls._create_(value, names, module=module, qualname=qualname, type=type, start=start)
  File "/home/vaultah/cpython/Lib/enum.py", line 342, in _create_
    classdict[member_name] = member_value
  File "/home/vaultah/cpython/Lib/enum.py", line 72, in __setitem__
    raise TypeError('Attempted to reuse key: %r' % key)
TypeError: Attempted to reuse key: 'a'

We can restrict stack trace entries to our code (in /home/vaultah/so/shadows/main.py).

import sys, enum
sys.excepthook = spotlight('/home/vaultah/so/shadows')
enum.Enum('Faulty', 'a a', module=__name__)

and

import sys, enum
sys.excepthook = shadow('/home/vaultah/cpython/Lib')
enum.Enum('Faulty', 'a a', module=__name__)

give the same result:

  File "/home/vaultah/so/shadows/main.py", line 22, in <module>
    enum.Enum('Faulty', 'a a', module=__name__)
TypeError: Attempted to reuse key: 'a'

There's a way to exclude all site directories (where 3rd party packages are installed - see site.getsitepackages)

import sys, site, jinja2
sys.excepthook = shadow(*site.getsitepackages())
jinja2.Template('{%}')
# jinja2.exceptions.TemplateSyntaxError: unexpected '}'
#     Generates ~30 lines, but will only display 4

Note: Don't forget to restore sys.excepthook from sys.__excepthook__. Unfortunately, you won't be able to "patch-restore" it using a context manager.

Wesson answered 9/10, 2015 at 15:43 Comment(1)
Perfect solution, no need to add global variables to all the project files unlike the accepted answer. ThanksTimeous
F
1

the traceback.extract_tb(tb) would return a tuple of error frames in the format(file, line_no, type, error_statement) , you can play with that to format the traceback. Also refer https://pymotw.com/2/sys/exceptions.html

import sys
import traceback

def handle_exception(ex_type, ex_info, tb):
    print ex_type, ex_info, traceback.extract_tb(tb)

sys.excepthook = handle_exception
Filomenafiloplume answered 6/10, 2015 at 16:25 Comment(2)
Could the guy who downvoted write a comment as what makes it worth a downvote , please ?Filomenafiloplume
I see 2 users downvoted this answer. Strictly speaking, it does not really answer the question, as in the information presented here is insufficient to solve OP's problem. Maybe you'd clarify what exactly does the asker need to do, rather than vaguely advising to "play with traceback.extract_tb".Wesson

© 2022 - 2024 — McMap. All rights reserved.