No Output From cProfile

Question

I'm trying to profile an application written using PySide and OpenCV and am getting strange behaviour with the profiler. I run my code using the following line:

python -m cProfile -o output.file repo/src/application_window.py

It tends to work for very short runs of the program. If I open it and load an image (it's an animation editor), output.file appears as expected. However, if I actually use the program (move through animation frames, etc.) there is no output. It generates no error messages or crash information - just nothing. The same problem occurs if I try to write the results to the console - sometimes it just produces no output.

Any thoughts on what could be causing this?

Edit: The call terminates as expected - but produces no output. It doesn't hang.

Edit2 - Here is the main function of application_window.py:

def main():
    a = QtGui.QApplication(sys.argv)
    editor = Editor(sys.argv[0])
    editor.show()

    sys.exit(a.exec_())

if __name__ == "__main__":
    main()

Editor is a fairly complicated QT Widget. So all application_window really does is start the QT event loop.

Lynettalynette

It terminates when expected, but with no output. – Lynettalynette

Does python application_window.py run on command line? – Formalin

Nope, it's a GUI application (it basically opens a QT Widget and displays that). – Lynettalynette

Did you ever get this to work? I have the same issue with a PySide6 application – Bumkin

Answer 1

There are a couple reasons why this might happen, but they all effectively boil down to something in the program making a call to FFI leading to some sort of shenanigans causing the process to exit early.

Most Likely Cause

Assuming that your program appears to be behaving normally, the most likely cause is that one of these functions is being called (documentation):

os.execl(path, arg0, arg1, ...)
os.execle(path, arg0, arg1, ..., env)
os.execlp(file, arg0, arg1, ...)
os.execlpe(file, arg0, arg1, ..., env)
os.execv(path, args)
os.execve(path, args, env)
os.execvp(file, args)
os.execvpe(file, args, env)

These functions from python's builtin os module have the effect of immediately exiting the current process and starting a new process in its place. They are surprisingly common in all kinds of programs due to their utility in creating launchers. For example, lets say you have a program that requires a session key to run. You might create a wrapper program that fetches the session key, then calls execve to start the main program while passing that session key as an argument or environment variable. The OS then gets to unload the resources associated with first process and the main process does not need to include the setup code. Another big benefit is that it lets you setup other tooling that runs the main program directly.

Unfortunately, they also have the effect of breaking any running cProfile/profile/pdb sessions. Since they all run in the same process as the code you are measuring, they have no way of persisting after the call. When executed, the process dies immediately without running any further code (including any registered atexit functions). cProfile could be updated to wrap these functions and write the profile before continuing, but it would be unrepresentative (and potentially misleading) of the full life cycle of the program.

Solution

You can solve this problem by using a profiler that runs in a separate process. The best option I currently know of is py-spy. It can profile multi-threaded programs and their child processes, so it should be able to handle this edge case. Just remember to call it with --subprocesses.

Allowable

Not a criticism of your answer --- I had a similar issue with a PySide6 application and neither cProfile or py-spy were able to profile my application, both had the same problem. – Bumkin

Answer 2

One possible explanation for this behavior is that the file gets created at the end of the run. The current directory of the process may at that point be a place you don't expect (or even a directory where the file can't be created). The quickest way to see if this is your problem is to leave out the -o output.file. cProfile then writes the profile to stdout. You can also try specifying an absolute path, e.g. -o /var/tmp/output.file.

Stull

This has was patched in python version 3.10.0a2. However, all prior versions are still effected by this issue. – Allowable