Get full traceback

Question

How can i get full traceback in the following case, including the calls of func2 and func functions?

import traceback

def func():
    try:
        raise Exception('Dummy')
    except:
        traceback.print_exc()

def func2():
    func()


func2()

When i run this, i get:

Traceback (most recent call last):
  File "test.py", line 5, in func
    raise Exception('Dummy')
Exception: Dummy

traceback.format_stack() is not what i want, as need traceback object to be passed to a third party module.

I am particularly interested in this case:

import logging


def func():
    try:
        raise Exception('Dummy')
    except:
        logging.exception("Something awful happened!")


def func2():
    func()


func2()

In this case i am getting:

ERROR:root:Something awful happened!
Traceback (most recent call last):
  File "test.py", line 9, in func
    raise Exception('Dummy')
Exception: Dummy

Sightly

@Nathan, please reread the question carefully. Full traceback is needed. – Sightly

See Generating full stack traces for exceptions in Python. blog post by Graham Dupleton. – Beckman

Please explain what you mean with "traceback.format_stack() is not what i want, as need traceback object to be passed to a third party module." Please show which code is from a third party and which code is from you. Please use appropriate method names. – Margeret

Maybe related: bugs.python.org/issue9427 "logging.error('...', exc_info=True) should display upper frames, too" – Margeret

@Margeret logger.log uses exc_info = sys.exc_info() which has the traceback in question, and this info is passed to Sentry handler which parses it to inspect local variables in each frame. Yes, I think the issue is the same. – Sightly

Addition of this feature to Python 3 is tracked in Add traceback.print_full_exception() issue – Beckman

Answer 1

As mechmind answered, the stack trace consists only of frames between the site where the exception was raised and the site of the try block. If you need the full stack trace, apparently you're out of luck.

Except that it's obviously possible to extract the stack entries from top-level to the current frame—traceback.extract_stack manages it just fine. The problem is that the information obtained by traceback.extract_stack comes from direct inspection of stack frames without creating a traceback object at any point, and the logging API requires a traceback object to affect traceback output.

Fortunately, logging doesn't require an actual traceback object, it requires an object that it can pass to the formatting routines of the traceback module. traceback doesn't care either—it only uses two attributes of the traceback, the frame and the line number. So, it should be possible to create a linked list of duck-typed faux-traceback objects and pass it off as the traceback.

import sys

class FauxTb(object):
    def __init__(self, tb_frame, tb_lineno, tb_next):
        self.tb_frame = tb_frame
        self.tb_lineno = tb_lineno
        self.tb_next = tb_next

def current_stack(skip=0):
    try: 1/0
    except ZeroDivisionError:
        f = sys.exc_info()[2].tb_frame
    for i in xrange(skip + 2):
        f = f.f_back
    lst = []
    while f is not None:
        lst.append((f, f.f_lineno))
        f = f.f_back
    return lst

def extend_traceback(tb, stack):
    """Extend traceback with stack info."""
    head = tb
    for tb_frame, tb_lineno in stack:
        head = FauxTb(tb_frame, tb_lineno, head)
    return head

def full_exc_info():
    """Like sys.exc_info, but includes the full traceback."""
    t, v, tb = sys.exc_info()
    full_tb = extend_traceback(tb, current_stack(1))
    return t, v, full_tb

With these functions in place, your code only requires a trivial modification:

import logging

def func():
    try:
        raise Exception('Dummy')
    except:
        logging.error("Something awful happened!", exc_info=full_exc_info())

def func2():
    func()

func2()

...to give the expected output:

ERROR:root:Something awful happened!
Traceback (most recent call last):
  File "a.py", line 52, in <module>
    func2()
  File "a.py", line 49, in func2
    func()
  File "a.py", line 43, in func
    raise Exception('Dummy')
Exception: Dummy

Note that the faux-traceback objects are fully usable for introspection—displaying local variables or as argument to pdb.post_mortem()—because they contain references to real stack frames.

Valdis

I know this info. If you look into my question again, you will se i use logging.exception which is a shortcut to what you wrote. – Sightly

@warwaruk You're quite right. The point of your question is that a try/except block shortens the stack trace, and you want the full one. I can't believe I've never noticed it before. – Valdis

@warwaruk I've now updated the answer to actually answer your question. If there's an easier way to do this, I'd like to know it as well. – Valdis

wow, you've done a lot of work. i was thinking in the same direction, but i could not believe there is no an easier way... I will accept your answer, but if someone posts a better answer - i might accept the other answer. Thank you! – Sightly

i just have one concern, though i haven't verified it. This logging.error(message, exc_info=smth) is intercepted and sent to sentry, which shows local variables in each calling frame. And seeing variables values is my primary goal with all this question. If i understand correctly, in you case because there is no real traceback - it won't work. So it might be better to look for another way to get the real traceback. – Sightly

@warwaruk I don't see why it wouldn't work—the mock tracebacks contain references to real stack frames. You could easily test it by calling pdb.post_mortem() with the traceback produced by full_exc_info(). – Valdis

@warwaruk Glad to hear. It would be nice if this feature were a part of Python... it's quite useful. Your question deserves many more votes than it got; thanks for bringing it up. – Valdis

Answer 2

This is based on user4815162342's answer, but a bit more minimalistic:

import sys
import collections

FauxTb = collections.namedtuple("FauxTb", ["tb_frame", "tb_lineno", "tb_next"])

def full_exc_info():
    """Like sys.exc_info, but includes the full traceback."""
    t, v, tb = sys.exc_info()
    f = sys._getframe(2)
    while f is not None:
        tb = FauxTb(f, f.f_lineno, tb)
        f = f.f_back
    return t, v, tb

It avoids throwing the dummy exception, at the cost of requiring the usage of sys._getframe(). It assumes being used in the except clause where the exception was caught, as it goes up two stack frames (full_exc_info and the function that calls full_exc_info – that would be the function that calls the raising code, and as such is already included in the original traceback).

This gives the same output as the code in user4815162342's answer.

If you don't mind the slight differences in formatting, you can also use

import logging

def func():
    try:
        raise Exception('Dummy')
    except:
        logging.exception("Something awful happened!", stack_info=True)

def func2():
    func()

func2()

which results in

ERROR:root:Something awful happened!
Traceback (most recent call last):
  File "test.py", line 5, in func
    raise Exception('Dummy')
Exception: Dummy
Stack (most recent call last):
  File "test.py", line 12, in <module>
    func2()
  File "test.py", line 10, in func2
    func()
  File "test.py", line 7, in func
    logging.exception("Something awful happened!", stack_info=True)

In this case, you'll get a trace from the try to the exception, and a second from the root call to the location of the logging call.

Formularize

In my mind, your second suggestion is the best answer. It lets you know what is the exception trace and what is simply the stack trace. The only thing is that I don't know if it would be "sent as a traceback object to a third party." – Fortuitous

Your first answer is also nice, because I don't like the idea of throwing a dummy exception, even if on purely theoretical grounds. – Fortuitous

Nice answer. In Python 3 it would probably be better to use a dataclass instead of named tuple (named tuples are way overused). Also note that stack_info is not present in Python 2. – Valdis

+1 bballdave025: I got here with maybe a bit of an XY problem, but the solution in my case seems to be just mylogger.exception("Something awful happened!", stack_info=True). (Docs.) – Mauritius

Answer 3

Stack trace is collected when exception bubbles up. So you should print traceback on top of desired stack:

import traceback

def func():
    raise Exception('Dummy')

def func2():
    func()


try:
    func2()
except:
    traceback.print_exc()

Uralaltaic

The trouble is that this changes semantics. If the OP wants func2 to continue running if func fails (and log the traceback), the exception must be handled in func2, not outside it. – Valdis

Answer 4

i have written a module that writes a more complete traceback

The module is here documentation is docs

(also you can get the module from pypi

sudo pip install pd

)

To catch and pring exceptions do the following:

import pd

try:
    <python code>
except BaseException:       
    pd.print_exception_ex( follow_objects = 1 )

The stack trace looks like this one here:

Exception: got it

#1  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 1) at      t  test_pd.py:29
Calls next frame at:
    raise Exception('got it') at: test_pd.py:29

#2  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 2) at test_pd.py:28
Calls next frame at:
    self.kuku2( depth - 1 ) at: test_pd.py:28

#3  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 3) at test_pd.py:28
Calls next frame at:
    self.kuku2( depth - 1 ) at: test_pd.py:28

#4  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 4) at test_pd.py:28
Calls next frame at:
    self.kuku2( depth - 1 ) at: test_pd.py:28

#5  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 5) at     test_pd.py:28
 Calls next frame at:
    self.kuku2( depth - 1 ) at: test_pd.py:28

#6  def kuku2(self = {'a': 42, 'b': [1, 2, 3, 4]}, depth = 6) at test_pd.py:28
Calls next frame at:
    self.kuku2( depth - 1 ) at: test_pd.py:28

#7  def main() at test_pd.py:44
Local variables:
n = {'a': 42, 'b': [1, 2, 3, 4]}
Calls next frame at:
    pd.print_exception_ex( follow_objects = 1 ) at: test_pd.py:44

follow_objects = 0 will not print out object content (with complex data structures follow_objects can take a lot of time).

Psychosurgery

File "F:\Python\lib\site-packages\pd\pdd.py", line 18 return value ^ TabError: inconsistent use of tabs and spaces in indentation – Chaoan

Answer 5

There's some more information that could be extracted from the traceback, and I sometimes prefer a neater, more 'logical' information instead of multi-line blob with files, line numbers and code snippetsgiven by traceback. Preferably one line should say all the essentials.

To achieve this I use following function:

def raising_code_info():
    code_info = ''
    try:    
        frames = inspect.trace()
        if(len(frames)):
            full_method_name = frames[0][4][0].rstrip('\n\r').strip()
            line_number      = frames[1][2]
            module_name      = frames[0][0].f_globals['__name__']
            if(module_name == '__main__'):
                module_name = os.path.basename(sys.argv[0]).replace('.py','')
            class_name = ''
            obj_name_dot_method = full_method_name.split('.', 1)
            if len(obj_name_dot_method) > 1:
                obj_name, full_method_name = obj_name_dot_method
                try:
                    class_name = frames[0][0].f_locals[obj_name].__class__.__name__
                except:
                    pass
            method_name = module_name + '.'
            if len(class_name) > 0:
                method_name += class_name + '.'
            method_name += full_method_name
            code_info = '%s, line %d' % (method_name, line_number)
    finally:
        del frames
        sys.exc_clear()
    return code_info

It gives . and line number, e.g.:

(example module name: test.py):

(line 73:)
def function1():
    print 1/0

class AClass(object):    
    def method2(self):
        a = []
        a[3] = 1

def try_it_out():
    # try it with a function
    try:
        function1()
    except Exception, what:
        print '%s: \"%s\"' % (raising_code_info(), what)

    # try it with a method
    try:
        my_obj_name = AClass()
        my_obj_name.method2()       
    except Exception, what:
        print '%s: \"%s\"' % (raising_code_info(), what)

if __name__ == '__main__':
     try_it_out()


test.function1(), line 75: "integer division or modulo by zero"
test.AClass.method2(), line 80: "list assignment index out of range"

Which might be slightly neater in some use-cases.

Foofaraw

I haven't checked your code, i trust you that it works. But in the question it is stated: "traceback.format_stack() is not what i want, as need traceback object to be passed to a third party module.". Anyway thank you for another code snippet. – Sightly