What cool hacks can be done using sys.settrace?

Question

I love being able to modify the arguments the get sent to a function, using settrace, like :

import sys

def trace_func(frame,event,arg):
    value = frame.f_locals["a"]
    if value % 2 == 0:
        value += 1
        frame.f_locals["a"] = value

def f(a):
    print a

if __name__ == "__main__":
    sys.settrace(trace_func)
    for i in range(0,5):
        f(i)

And this will print:

1
1
3
3
5

What other cool stuff can you do using settrace?

Yak

Answer 1

I would strongly recommend against abusing settrace. I'm assuming you understand this stuff, but others coming along later may not. There are a few reasons:

1. Settrace is a very blunt tool. The OP's example is a simple one, but there's practically no way to extend it for use in a real system.

2. It's mysterious. Anyone coming to look at your code would be completely stumped why it was doing what it was doing.

3. It's slow. Invoking a Python function for every line of Python executed is going to slow down your program by many multiples.

4. It's usually unnecessary. The original example here could have been accomplished in a few other ways (modify the function, wrap the function in a decorator, call it via another function, etc), any of which would have been better than settrace.

5. It's hard to get right. In the original example, if you had not called f directly, but instead called g which called f, your trace function wouldn't have done its job, because you returned None from the trace function, so it's only invoked once and then forgotten.

6. It will keep other tools from working. This program will not be debuggable (because debuggers use settrace), it will not be traceable, it will not be possible to measure its code coverage, etc. Part of this is due to lack of foresight on the part of the Python implementors: they gave us settrace but no gettrace, so it's difficult to have two trace functions that work together.

Trace functions make for cool hacks. It's fun to be able to abuse it, but please don't use it for real stuff. If I sound hectoring, I apologize, but this has been done in real code, and it's a pain. For example, DecoratorTools uses a trace function to perform the magic feat of making this syntax work in Python 2.3:

# Method decorator example
from peak.util.decorators import decorate

class Demo1(object):
    decorate(classmethod)   # equivalent to @classmethod
    def example(cls):
        print "hello from", cls

A neat hack, but unfortunately, it meant that any code that used DecoratorTools wouldn't work with coverage.py (or debuggers, I guess). Not a good tradeoff if you ask me. I changed coverage.py to provide a mode that lets it work with DecoratorTools, but I wish I hadn't had to.

Even code in the standard library sometimes gets this stuff wrong. Pyexpat decided to be different than every other extension module, and invoke the trace function as if it were Python code. Too bad they did a bad job of it.

</rant>

Agreed

sys.gettrace exists since Python 2.6 (docs.python.org/library/sys.html#sys.gettrace), but because of the way settrace is designed, with the handler returning a new handler function on each call, it doesn't really help. I completely agree with your main points -- settrace is unusable for anything other than (slow) debugging. – Rochelle

This answer was written in 2009 and I should say that not all of the points raised by the author apply anymore. sys.settrace() and other debuggers can coexist depending on the implementation (such as ipdb.set_trace()). Also there is gettrace() function implemented...Also depending on the implementation, your code doesn't necessarily have to run "slow by many multiples". It will be never be faster, yes, but that's still a gross overgeneralization. – Presser

@Presser that is not an overgeneralization: adding code is always slower. Moreover, debugging tools are not made for performance, but for debugging. Finally, using debugging tools as functional code is one of the worst practices I've ever met. – Palate

@Palate I clearly said the code will run slower but not necessarily "slow by many multiples". Moreover, I was criticising the absolute statement that settrace is unusable for anything other than slow debugging. There are certainly practices where it is useful besides that, and they don't have to have anything to do with debugging. It's simply a tool to keep a track of function calls and files. I've seen it used an automatic archival tool to pair a rapidly changing codebase containing an analysis pipeline with its output, in order to ensure easy and accurate reproduction of the same output. – Presser

@aarslan, well, if your point is sys.settrace() can be used for code coverage tools or testing, I do agree, but you're not saying it explicitly. My point is always use sys.settrace() for meta- programming, not for programming, and so is @Ned's. – Palate

Answer 2

I made a module called pycallgraph which generates call graphs using sys.settrace().

Mesics

I can't see the added value compared to gprof2dot which produces exactly the same graphs for years (since 2007-03-30 exactly see the GitHub project), except that it adds timing info about each called function (self time, total time, number of calls). Nice work though. – Palate

Answer 3

Of course, code coverage is accomplished with the trace function. One cool thing we haven't had before is branch coverage measurement, and that's coming along nicely, about to be released in an alpha version of coverage.py.

So for example, consider this function:

def foo(x):
    if x:
        y = 10
    return y

if you test it with this call:

assert foo(1) == 10

then statement coverage will tell you that all the lines of the function were executed. But of course, there's a simple problem in that function: calling it with 0 raises a UnboundLocalError.

Branch measurement would tell you that there's a branch in the code that isn't fully exercised, because only one leg of the branch is ever taken.

Agreed

Answer 4

For example, get the memory consumption of Python code line-by-line: http://pypi.python.org/pypi/memory_profiler

Hague

Answer 5

One latest project that uses settrace heavily is PySnooper

It helps new programmers to trace/log/monitor their program output. Cheers!

Galengalena

Answer 6

A really cool project using settrace adds the defer statement from go to the language in a cool way. It allows code like this:

def foo():
  print(", world!") in defer
  print("Hello", end="")
  # do something that might fail...
  assert 1 + 1 == 3

This will output:

$ python foo.py
Hello, World!
Traceback (most recent call last):
  File "foo.py", line 7, in <module>
    assert 1 + 1 == 3
AssertionError

https://github.com/yasyf/python-defer

Demirep

Answer 7

The python debugger Pdb uses sys.settrace to analyse lines to debug.

Here's an c optimization/extension for pdb that also uses sys.settrace

https://bitbucket.org/jagguli/cpdb

Geelong

Answer 8

I don't have an exhaustively comprehensive answer but one thing I did with it, with the help of another user on SO, was create a program that generates the trace tables of other Python programs.

Legpull