Python assert -- improved introspection of failure?

Question

This is a rather useless assertion error; it does not tell the values of the expression involved (assume constants used are actually variable names):

$ python -c "assert 6-(3*2)"
[...]
AssertionError

Is there a better assert implementation in Python that is more fancy? It must not introduce additional overhead over execution (except when assert fails) .. and must turn off if -O flag is used.

Edit: I know about assert's second argument as a string. I don't want to write one .. as that is encoded in the expression that is being asserted. DRY (Don't Repeat Yourself).

Annecorinne

Answer 1

Install your of function as sys.excepthook -- see the docs. Your function, if the second argument is AssertionError, can introspect to your heart's contents; in particular, through the third argument, the traceback, it can get the frame and exact spot in which the assert failed, getting the failing exception through the source or bytecode, the value of all relevant variables, etc. Module inspect helps.

Doing it in full generality is quite a piece of work, but depending on what constraints you're willing to accept in how you write your asserts it can be lightened substantially (e.g. restricting them to only local or global variables makes introspection easier than if nonlocal variables of a closure could be involved, and so forth).

Darin

Good. Now is there a Python library for this .. or do I have to write my own? :-) (I probably won't .. as this is a low-prio task for me) – Annecorinne

Unfortunately I don't know of existing Python libraries doing all of this, except ones oriented to testing (which might have to be adapted for the purpose of using them on production code). – Darin

Answer 2

You can attach a message to an assert:

assert 6-(3*2), "always fails"

The message can also be built dynamically:

assert x != 0, "x is not equal to zero (%d)" % x

See The assert statement in the Python documentation for more information.

Minutiae

Of course, I knew this. I don't want to write one as that is encoded in the expression that is being asserted. DRY. – Annecorinne

I see what you mean. I don't believe Python has a way to do this. – Minutiae

Node's assert shows the actual value vs. the expected one. This is a really basic features,. Has Python not implemented it in the intervening 14 years? – Rosol

Answer 3

As @Mark Rushakoff said nose can evaluate failed asserts. It works on the standard assert too.

# test_error_reporting.py
def test():
    a,b,c = 6, 2, 3
    assert a - b*c

nosetests' help:

$ nosetests --help|grep -B2 assert
  -d, --detailed-errors, --failure-detail
                        Add detail to error output by attempting to evaluate
                        failed asserts [NOSE_DETAILED_ERRORS]

Example:

$ nosetests -d
F
======================================================================
FAIL: test_error_reporting.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "..snip../site-packages/nose/case.py", line 183, in runTest
    self.test(*self.arg)
  File "..snip../test_error_reporting.py", line 3, in test
    assert a - b*c
AssertionError:
    6,2,3 = 6, 2, 3
>>  assert 6 - 2*3


----------------------------------------------------------------------
Ran 1 test in 0.089s

FAILED (failures=1)

Donahue

The question is regarding the use of assert in application code (which is directly invoked by the user, for eg., ./foo.py .. or clicking on 'foo.pyw' on Windows Explorer), and not test code .. for which I am actually happy with py.test's assert output. – Annecorinne

@srid: In this case write: __debug__ and your_fancy_assert(expression) -- no overhead on '-O'. – Donahue

That's sounds interesting; too bad Python doesn't have a macro feature. – Annecorinne

@SridharRatnakumar: it has (now) macropy – Donahue

For the introspection to work, you also need to leave out the optional msg parameter. – Gregg

Answer 4

The nose testing suite applies introspection to asserts.

However, AFAICT, you have to call their asserts to get the introspection:

import nose
def test1():
    nose.tools.assert_equal(6, 5+2)

results in

C:\temp\py>C:\Python26\Scripts\nosetests.exe -d test.py
F
======================================================================
FAIL: test.test1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line
183, in runTest
    self.test(*self.arg)
  File "C:\temp\py\test.py", line 3, in test1
    nose.tools.assert_equal(6, 5+2)
AssertionError: 6 != 7
>>  raise self.failureException, \
          (None or '%r != %r' % (6, 7))

Notice the AssertionError there. When my line was just assert 6 == 5+2, I would get:

C:\temp\py>C:\Python26\Scripts\nosetests.exe -d test.py
F
======================================================================
FAIL: test.test1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\nose-0.11.1-py2.6.egg\nose\case.py", line
183, in runTest
    self.test(*self.arg)
  File "C:\temp\py\test.py", line 2, in test1
    assert 6 == 5 + 2
AssertionError:
>>  assert 6 == 5 + 2

Also, I'm not sure offhand if their asserts are skipped with -O, but that would be a very quick check.

Pristine

Good enough for test cases, but for production code .. there is function call overhead (even with -O option) – Annecorinne

Ordinary asserts work too. See stackoverflow.com/questions/1308607/… – Donahue

Answer 5

I coded a replacement for sys.excepthook (which is called for any unhandled exception) which is a bit more fancy than the standard one. It will analyze the line where the exception occured and print all variables which are referred to in this line (it does not print all local variables because that might be too much noise - also, maybe the important var is global or so).

I called it py_better_exchook (perfect name) and it's here.

Example file:

a = 6

def test():
    unrelated_var = 43
    b,c = 2, 3
    assert a - b*c

import better_exchook
better_exchook.install()

test()

Output:

$ python test_error_reporting.py 
EXCEPTION
Traceback (most recent call last):
  File "test_error_reporting.py", line 12, in <module>
    line: test()
    locals:
      test = <local> <function test at 0x7fd91b1a05f0>
  File "test_error_reporting.py", line 7, in test
    line: assert a - b*c
    locals:
      a = <global> 6
      b = <local> 2
      c = <local> 3
AssertionError

There are a few other alternatives:

• (Presented here) https://github.com/albertz/py_better_exchook/
• https://github.com/patrys/great-justice
• Nose does something similar for assertion failures, see here.
• IPython has something similar (this). Do this: from IPython.core import ultratb; sys.excepthook = ultratb.VerboseTB().
• Ka-Ping Yee's "cgitb.py", which is part of Python, see here, code here.

Matildamatilde

Can this be used in the same way as import traceback; traceback.print_stack(), ie, printing then continuing execution? (I tried looking at the code but it was beyond me.) Huge thanks, very glad to discover this, been wanting to get better exception reports for years! – Lepidote

@Lepidote Yes sure. You just catch the exception in try: ... except Exception: ..., then you print the exception with all info by sys.excepthook(*sys.exc_info()), and then it continues. – Matildamatilde

Thank you! Is artificially raising an exception, eg, try:raise Exception();except Exception:sys.excepthook(*sys.exc_info()) the best way even if there's no exception and I'm just looking for a dump of state at a point in working code? – Lepidote

On testing this, it just seems to report the try block scope at the lowest level, nothing higher and no variables outside the lowest try block (so just Exception in this case). – Lepidote

If you don't have an exception, you don't need to raise a dummy one. You could directly call better_exchook.print_tb(None), which will print the current traceback. – Matildamatilde

Wow! I'm working on a tangled performance issue and this will be invaluable, thanks! – Lepidote

Btw, printing the stacktrace / exceptions is usually helpful on unexpected exceptions. For debugging, if possible, it is usually better to use a real debugger in an IDE. E.g. I can recommend PyCharm (the free community version is just fine). You just put a breakpoint where ever you want, and then you get all information you want. – Matildamatilde

Thanks very much, I do use PyCharm but this only runs inside a couple Docker containers (and I find the PyCharm Docker integration clunky) and it is a bit of a forensic exercise figuring out where the flow is going in a long batch processing sequence, so this is perfect. – Lepidote

10/10 for mentioning alternatives to your own library! – Rhoda

Answer 6

It sounds like what you really want to do is to set up a debugger breakpoint just before the assert and inspect from your favorite debugger as much as you like.

Penrod

That's a workaround, but it's rather dumb for assert to not show the actual value on value. – Rosol

Answer 7

Add a message to your assertion, which will be displayed if the assertion fails:

$ python -c "assert 6-(3*2), '6-(3*2)'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: 6-(3*2)

The only way I can think of to provide this automatically would be to contain the assertion in a procedure call, and then inspect the stack to get the source code for that line. The additional call would, unfortunately, introduce overhead into the test and would not be disabled with -O.

Brother

Precisely. It is disabling this 'introspection' on -O that is the key to the question. – Annecorinne

.. BUT this is not an overhead if this function is called only during assertion errors (not assertion calls). – Annecorinne