cProfile saving data to file causes jumbles of characters

Question

I am using cProfile on a module named bot4CA.py so in the console I type:

python -m cProfile -o thing.txt bot4CA.py

After the module runs and exits, it creates a file named thing.txt and when I open it, there is some information there, and the rest is a jumble of characters instead of a neatly organized file of data which is what I want. Does any one know how to use cProfile and end up with neatly organized table of data like when using it normally on command line, except in a file? Here's an example of some of the data in the .txt file:

{(   s)   build\bdist.win32\egg\colorama\winterm.pyi'   t      reset_all(   i   i   gpàÂs% ?geOÙHÌœE?{(   s-   build\bdist.win32\egg\colorama\ansitowin32.pyi¥

What I really want is what happens when you invoke cProfile.run() which results in a neatly organized table printed showing the execution times of all the functions except instead of printed, saved in a file as this program is fairly large and runs a lot of functions.

Willner

Answer 1

You should use the pstats module to parse this file and extract information in user-friendly format from it. For example:

import pstats
p = pstats.Stats('thing.txt')
p.sort_stats('cumulative').print_stats(10)

It's all in the documentation, of course. Go over the "instant user's manual" in there, it explains everything.

Oscoumbrian

How would I be able to do that using command line as with: python -m cProfile -o thing.txt bot4CA.py. The reason why is because cProfile doesn't seem to want to work with running entire scripts. – Willner

well, now I have a second problem, this program uses twisted and the data only goes as deep as twisted's main loop, rather than the methods that twisted calls in the script – Willner

@joseph: the linked doc explains command-line usage, although it's relatively limited in the reporting options it offers – Oscoumbrian

What about the twisted problem where it only gives the times for the twisted main loop or something on that level, instead of the methods that the twisted Framework calls like dataRecieved? – Willner

@joseph: I have no idea - never tried profiling Twisted. You can open a separate question or ask in some Twisted forums/lists – Oscoumbrian

Unfortunately, there's nothing in the linked documentation that describes what the file format is, or what one can do with it... – Dramatist

@ChrisDodd: the format is internal, and specifically not documented. You're supposed to use pstats to work with it. This answers the question, which didn't inquire about the internal format, but rather how to make sense of it. – Oscoumbrian

Use dump_stats to save it to a file. – Pachysandra

In the documentation I didn't see anything like this, but your code was very useful. Thanks! – Featherhead

Answer 2

to dump the stats driectly:

echo 'stats' | python3 -m pstats path/to/cprofile_output_file

---

pstats also has a shell

$ python3 -m pstats path/to/cprofile_output_file

within we can issue stats or sort commands like so:

$ python3 -m pstats path/to/cprofile_output_file
Welcome to the profile statistics browser.
prof.txt% sort cumtime
prof.txt% reverse
prof.txt% stats

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 63:1(<module>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

prof.txt% ?

Documented commands (type help <topic>):
========================================
EOF  add  callees  callers  help  quit  read  reverse  sort  stats  strip

---

a micro-feature I enjoyed here is I get to reverse the sort order natively <3

echo -e 'sort cumtime\nreverse\nstats' | python3 -m pstats path/to/cprofile_output_file

Reft

This is a new answer on a very old question, but in my opinion it is much more helpful than the accepted answer because it matches the OP's command line cProfile usage. Annoyingly, the official docs don't demonstrate parsing the output file anywhere near demonstrating how to generate it. +1 – Goober

also: echo -e 'sort tottime\nreverse\nstats' | python3 -m pstats <(python3 -m cProfile -o /dev/stdout ./myscript.py myargs) – Reft

This was helpful. On my windows machine, the first part, the echo statement, is without quotes. Use (e.g.) echo stats | python -m pstats path\to\profile_output – Metaphrast

Answer 3

The other answers are more powerful and flexible, but if you just want to get a quick output, use > instead of -o. That will save the report in plain text format.

python -m cProfile myscript.py > cprofile.txt
python -m cProfile bot4CA.py > thing.txt

Whitmer

Note that this will also save all the output of your program also into the same file – Disendow