File "<string>" traceback with line preview

Question

I'm making a couple of functions via exec, that could possibly error. However when Python errors out it doesn't display the line that the error occurred on.

For example using:

fn_str = '''\
def fn():
    raise Exception()
'''

globs = {}
exec(fn_str, globs)
fn = globs['fn']

fn()

Gives us the output:

Traceback (most recent call last):
  File "...", line 10, in <module>
    fn()
  File "<string>", line 2, in fn
Exception

---

If we however, don't use eval. Then we get the line that the program errored on:

def fn():
    raise Exception()

fn()

Traceback (most recent call last):
  File "...", line 4, in <module>
    fn()
  File "...", line 2, in fn
    raise Exception()
Exception

---

I've looked into using the __traceback__, however I couldn't find a way to add to the traceback under the 'File' line. And so the best I could get was this:

fn_str = '''\
def fn():
    try:
        raise Exception()
    except BaseException as e:
        tb = e.__traceback__
        if 1 <= tb.tb_lineno <= len(fn_lines):
            e.args = ((e.args[0] if e.args else '') + ' - ' + fn_lines[tb.tb_lineno - 1].strip(),)
        raise
'''

globs = {'fn_lines': fn_str.split('\n')}
exec(fn_str, globs)
fn = globs['fn']

fn()

Traceback (most recent call last):
  File "...", line 16, in <module>
    fn()
  File "<string>", line 3, in fn
Exception:  - raise Exception()

The biggest problem with this is if the eval calls other code, it becomes confusing where the - raise Exception() comes from.

---

Is there a way to make the eval code provide the line that it errored on?

Cumine

Answer 1

The missing lines are a symptom of the fact that Python cannot find a file on disk named <string>, which is the filename built into your compiled snippet of code. (Though if you create a file with exactly that name, Python will print lines from it!)

Approach 1. You can catch exceptions yourself, whether at the top level of your application or elsewhere, and instead of letting the default builtin traceback routine fire you can call the Standard Library routine traceback.print_exc() which pulls lines from the Standard Library module linecache. Because the linecache cache is a simple public Python dictionary, you can pre-populate it with the source code it needs to print. See:

Why does the Python linecache affect the traceback module but not regular tracebacks?

The resulting code:

import linecache
import traceback

source = 'print("Hello, world" + 1)'
source_name = 'Example'
lines = source.splitlines(True)
linecache.cache[source_name] = len(source), None, lines, source_name
compiled = compile(source, source_name, 'exec')
try:
    eval(compiled)
except Exception:
    traceback.print_exc()

Approach 2. You can also avoid the indirection of populating a global cache by simply taking charge of printing the exception yourself: you can have Python return the traceback data as a list of tuples, step through them adding the missing lines, and then finally print them as usual.

Here is a fill_in_lines() function that fills out the traceback with the missing information, in a small program that prints the full traceback:

import sys
import traceback

def fill_in_lines(frames, source_name, source):
    lines = source.splitlines()
    for filename, line_number, function_name, text in frames:
        if filename == source_name:
            text = lines[line_number - 1]
        yield filename, line_number, function_name, text

source = 'print("Hello, world" + 1)'
source_name = 'Example'
compiled = compile(source, source_name, 'exec')
try:
    eval(compiled)
except Exception as e:
    _, _, tb = sys.exc_info()
    frames = traceback.extract_tb(tb)
    frames = fill_in_lines(frames, source_name, source)

    print('Traceback (most recent call last):')
    print(''.join(traceback.format_list(frames)), end='')
    print('{}: {}'.format(type(e).__name__, str(e)))

I am able to use the fancy name “Example” here because I set it using compile(). In your case you would instead want to pass the bare string '<string>' as the source_name argument.

Velours

Interesting. I've verified that this works but I'm off to bed. Thank you for the answer, I'll look into it more tomorrow – Cumine

I'm glad it worked! Upon waking this morning I thought, "so why didn't it work when I pre-populated the line cache," and did a quick search, and found the Stack Overflow answer that already tackles the subject. No idea why I didn't find it yesterday! I have updated my answer to include a link to that approach too. – Velours

Answer 2

Here is a helper function that puts all of these ideas together, to automatically write a file and rewrite the backtrace to point to it when you have an exception that has a frame:

import traceback
import sys
from types import TracebackType
import tempfile
import contextlib
import inspect

@contextlib.contextmanager
def report_compile_source_on_error():
    try:
        yield
    except Exception as exc:
        tb = exc.__traceback__

        # Walk the traceback, looking for frames that have
        # source attached
        stack = []
        while tb is not None:
            filename = tb.tb_frame.f_code.co_filename
            source = tb.tb_frame.f_globals.get("__compile_source__")

            if filename == "<string>" and source is not None:
                # Don't delete the temporary file so the user can expect it
                with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
                    f.write(source)
                # Create a frame.  Python doesn't let you construct
                # FrameType directly, so just make one with compile
                frame = tb.tb_frame
                code = compile('__inspect_currentframe()', f.name, 'eval')
                # Python 3.8 only.  In earlier versions of Python
                # just have less accurate name info
                if hasattr(code, 'replace'):
                    code = code.replace(co_name=frame.f_code.co_name)
                fake_frame = eval(
                    code,
                    frame.f_globals,
                    {
                        **frame.f_locals,
                        '__inspect_currentframe': inspect.currentframe
                    }
                )
                fake_tb = TracebackType(
                    None, fake_frame, tb.tb_lasti, tb.tb_lineno
                )
                stack.append(fake_tb)
            else:
                stack.append(tb)

            tb = tb.tb_next

        # Reconstruct the linked list
        tb_next = None
        for tb in reversed(stack):
            tb.tb_next = tb_next
            tb_next = tb

        raise exc.with_traceback(tb_next)

Full file with a test at https://gist.github.com/ezyang/ed041c0302d4c2a63cc51be5b10660da

Irresistible

Answer 3

This is a repost of vaultah's deleted answer.

That happens when the interpreter is unable to find the source code for that line for whatever reason. This is the case for built-in modules, compiled files, exec strings, etc. More specifically, in the traceback you can see that the filename for the fn's code object is set to <string>

File "<string>", line 2, in fn

Because <string> is not a valid filename, the reference to the source code is lost.

One option is to create a temporary file, write fn_str in there, compile fn_str to set the filename, execute the compiled code, and finally call the function. Note that you'll need to keep the file alive at least until the source lines are cached by the traceback-printing facility

from tempfile import NamedTemporaryFile
import traceback

with NamedTemporaryFile('w') as temp:
    code = compile(fn_str, temp.name, 'exec')
    print(fn_str, file=temp, flush=True)
    globs = {}
    exec(code, globs)
    fn = globs['fn']
    try:
        fn()
    except:
        traceback.print_exc()

prints

Traceback (most recent call last):
  File "test.py", line 16, in <module>
    fn()
  File "/tmp/tmp9q2bogm6", line 2, in fn
    raise Exception()
Exception

Since we already create a "real" file, we can delegate the compilation and execution of code to runpy.run_path:

from tempfile import NamedTemporaryFile
import runpy, traceback

with NamedTemporaryFile('w') as temp:
    print(fn_str, file=temp, flush=True)
    fn = runpy.run_path(temp.name)['fn']
    try:
        fn()
    except:
        traceback.print_exc()

Cumine