Profiling a python 3.6 module from the command line

Question

I have a python project where I execute the app as a module using the -m flag. So something like:

python -m apps.validate -i input.mp4

Now, I want to profile it using the command line. So the inline examples suggest invoking cProfile itself a module. However, I cannot do something like:

python -m cProfile apps.validate -i input.mp4

However, this results in the error "No such file or directory". I cannot just go to the apps directory and launch validate.py due to relative imports.

Is there a way to profile a module on the command line?

Thacker

There's a "New in version 3.7" note in the cProfile online documentation that says a -m option was added to cProfile in that version. This is in addition to the Python interpreter's own -m option, This means that something like python -m cProfile -m apps.validate -i input.mp4 ought to work (if you're using Python 3.7+). – Hermaphrodite

I see. Unfortunately, my stuff needs python 3.6 as some of the other libraries are not working with 3.7. I guess I will need to use it in code. – Thacker

You could look at the source code for profile (and cProfile) and see how support for the new -m option was added. It might even be possible to use that version of it with an earlier version of the Python interpreter (depending on what other changes were made). – Hermaphrodite

Answer 1

Instead of running cProfile in shell, maybe you can use cProfile in your python script by adding some code in apps.validate or creating a new script and import apps.validate like this. Maybe some typo below :)

import cProfile
import sys

def run_validate(args): 
    # run your apps.validate code with shell arguments args here
    pass

if __name__ == '__main__':
    pr = cProfile.Profile()
    pr.enable()
    run_validate(*sys.argv)
    pr.disable()
    pr.print_stats()

then just run the original script: python -m apps.validate -i input.mp4

Luba

pr.dump_stats(filename) would save it to a file. – Mord