How can you get the call tree with Python profilers?

Question

I used to use a nice Apple profiler that is built into the System Monitor application. As long as your C++ code was compiled with debug information, you could sample your running application and it would print out an indented tree telling you what percent of the parent function's time was spent in this function (and the body vs. other function calls).

For instance, if main called function_1 and function_2, function_2 calls function_3, and then main calls function_3:

main (100%, 1% in function body):
    function_1 (9%, 9% in function body):
    function_2 (90%, 85% in function body):
        function_3 (100%, 100% in function body)
    function_3 (1%, 1% in function body)

I would see this and think, "Something is taking a long time in the code in the body of function_2. If I want my program to be faster, that's where I should start."

How can I most easily get this exact profiling output for a Python program?

I've seen people say to do this:

import cProfile, pstats
prof = cProfile.Profile()
prof = prof.runctx("real_main(argv)", globals(), locals())
stats = pstats.Stats(prof)
stats.sort_stats("time")  # Or cumulative
stats.print_stats(80)  # 80 = how many to print

But it's quite messy compared to that elegant call tree. Please let me know if you can easily do this, it would help quite a bit.

Enchanting

Answer 1

Check out this library http://pycallgraph.slowchop.com/ for call graphs. It works really well. If you want to profile specific functions, check out http://mg.pov.lt/blog/profiling.html

This is a result from the profilehooks module.

Pluvious

Do you know if this package can do elapsed time as well as number of function calls? For instance, I wouldn't mind calling __hash__() thousands of times, but calling sorted() repeatedly on large lists might be quite slow. – Enchanting

You should just try them both out. They basically do the same thing in different ways with different outputs. – Pluvious

+1, the package is along the lines of what I had in mind and was quick and easy to install and call. Do you mind telling me how you got the above figure? I made a graph (using pycallgraph.start_trace() and pycallgraph.make_dot_graph('test.jpg', format='jpg', tool='neato') ), but it's far too crowded and lots of functions are obscured. – Enchanting

@Oliver: The above graph was created with the profilehooks module, not the callgraph. – Pluvious

It looks like a screenshot of RunSnakeRun to me. profilehooks is just a bunch of decorators that wrap cProfiler for easier usage. – Firecure

Note: pycallgraph is now unmainainted – Frizzy

Good to know. Note: the latest version is 1.0.1 which was released on 2013-09-17, and announcement in this message from 2018 at "Retiring Projects | Gerald Kaszuba" geraldkaszuba.com/retiring-projects – Shofar

pycallgraph3 at github.com/vmdesenvolvimento/pycallgraph3 a fork of pycallgraph was maintained up to July 2020. Thats the date of the most recent checkin. – Eichelberger

Answer 2

I just stumbled on this as well, and spent some time learning how to generate a call graph (the normal results of cProfile is not terribly informative). Future reference, here's another way to generate a beautiful call-tree graphic with cProfile + gprof2dot + graphViz.

———————

1. Install GraphViz: http://www.graphviz.org/Download_macos.php
2. easy_install gprof2dot

3. Run profile on the code.

   python -m cProfile -o myLog.profile <myScript.py> arg1 arg2 ...
   

4. Run gprof2dot to convert the call profile into a dot file

   gprof2dot -f pstats myLog.profile -o callingGraph.dot
   

5. Open with graphViz to visualize the graph

Here's what the end result would look like! Graph is color-coded- red means higher concentration of time.

Sherasherar

xdot is a nice interactive viewer for .dot files that lets you zoom in on stuff. – Firecure

Can display the .dot file on Unix with a shell command like "dot callingGraph.dot -Tpng | display" in place of step 5. – Sadfaced

Answer 3

I recently wanted the same thing, so I took a stab at implementing one myself.

The project's on GitHub, https://github.com/joerick/pyinstrument

Here's how you would use it:

from pyinstrument import Profiler

profiler = Profiler()
profiler.start()

# Code you want to profile

profiler.stop()

print(profiler.output_text())

Unmistakable

It seems nice, here is one that is more a generic library: github.com/vmprof/vmprof-python – Sporadic

Answer 4

The gprof2dot approach extracts all information nicely, so I'm a fan. However, sometimes I want to look at timing data in a call tree, so I created tuna.

Install with

pip install tuna

and display your profile with

tuna program.prof

Mejia

Love this profiler !! It was so easy to find my bottleneck functions and it was as per my intuition. So 5 stars for the Tuna tool – Tasman

Answer 5

You can use SnakeViz:

pip install snakeviz

Create profile file:

python -m cProfile -o program.prof my_program.py

Generate the interactive graph:

snakeviz program.prof

Yeargain