accurately measure time python function takes

Question

I need to measure the time certain parts of my program take (not for debugging but as a feature in the output). Accuracy is important because the total time will be a fraction of a second.

I was going to use the time module when I came across timeit, which claims to avoid a number of common traps for measuring execution times. Unfortunately it has an awful interface, taking a string as input which it then eval's.

So, do I need to use this module to measure time accurately, or will time suffice? And what are the pitfalls it refers to?

Thanks

Jorgensen

Accuracy? Sub-second? Since most OS's have very flexible scheduling, these two doesn't work together. Sub-second events cannot be guaranteed to be scheduled consistently. You'll have huge variability in the way your process is scheduled. What are you really trying to do? – Konya

Wouldn't the python module "profile" provide the result you need ? – Nicknickel

Answer 1

According to the Python documentation, it has to do with the accuracy of the time function in different operating systems:

The default timer function is platform dependent. On Windows, time.clock() has microsecond granularity but time.time()‘s granularity is 1/60th of a second; on Unix, time.clock() has 1/100th of a second granularity and time.time() is much more precise. On either platform, the default timer functions measure wall clock time, not the CPU time. This means that other processes running on the same computer may interfere with the timing ... On Unix, you can use time.clock() to measure CPU time.

To pull directly from timeit.py's code:

if sys.platform == "win32":
    # On Windows, the best timer is time.clock()
    default_timer = time.clock
else:
    # On most other platforms the best timer is time.time()
    default_timer = time.time

In addition, it deals directly with setting up the runtime code for you. If you use time you have to do it yourself. This, of course saves you time

Timeit's setup:

def inner(_it, _timer):
    #Your setup code
    %(setup)s
    _t0 = _timer()
    for _i in _it:
        #The code you want to time
        %(stmt)s
    _t1 = _timer()
    return _t1 - _t0

Python 3:

Since Python 3.3 you can use time.perf_counter() (system-wide timing) or time.process_time() (process-wide timing), just the way you used to use time.clock():

from time import process_time

t = process_time()
#do some stuff
elapsed_time = process_time() - t

The new function process_time will not include time elapsed during sleep.

Python 3.7+:

Since Python 3.7 you can also use process_time_ns() which is similar to process_time()but returns time in nanoseconds.

Manstopper

timeit.default_timer is time.perf_counter on Python 3.3+ i.e., you could use default_timer on all versions. – Clypeus

for the record if you're using this in a script I had to do import time instead of import time.process_time, or from time import process_time on 3.4, or maybe I'm doing something wrong ;) – Adynamia

@Adynamia You're not wrong at all. This answer explains (a bit) why that is: you need to clarify whether time is referring to the module, or the function inside it. – Plumbery

@hellobenallan nice, thanks for confirming and making the edits. I didn't go ahead and do that because I wasn't entirely sure :) – Adynamia

The quote from the docs seems to contradict the comments in timeit.py's code. The former seems to say that time.clock() should be used (on Unix) to avoid timing other things going on with the scheduler, etc. and to get an accurate CPU time; but the latter seems to say to use time.time() for that purpose. Which should I use on Unix (OS X) to get CPU time? – Obscurant

In Python 3.7 you can use process_time_ns() which is similar to process_time but returns time in nanoseconds. – Amoreta

Answer 2

You could build a timing context (see PEP 343) to measure blocks of code pretty easily.

from __future__ import with_statement
import time

class Timer(object):
    def __enter__(self):
        self.__start = time.time()

    def __exit__(self, type, value, traceback):
        # Error handling here
        self.__finish = time.time()

    def duration_in_seconds(self):
        return self.__finish - self.__start

timer = Timer()

with timer:
    # Whatever you want to measure goes here
    time.sleep(2)

print timer.duration_in_seconds()    

Viceregal

this looks simple enough for cross-grained measures. But for fine-grained atomic operations like those described in the questions (where milliseconds matter),, I am not sure. – Overland

Answer 3

The timeit module looks like it's designed for doing performance testing of algorithms, rather than as simple monitoring of an application. Your best option is probably to use the time module, call time.time() at the beginning and end of the segment you're interested in, and subtract the two numbers. Be aware that the number you get may have many more decimal places than the actual resolution of the system timer.

Legal

yeah that was what I thought of originally, before I saw the timeit module – Jorgensen

Answer 4

I was annoyed too by the awful interface of timeit so i made a library for this, check it out its trivial to use

from pythonbenchmark import compare, measure
import time

a,b,c,d,e = 10,10,10,10,10
something = [a,b,c,d,e]

def myFunction(something):
    time.sleep(0.4)

def myOptimizedFunction(something):
    time.sleep(0.2)

# comparing test
compare(myFunction, myOptimizedFunction, 10, input)
# without input
compare(myFunction, myOptimizedFunction, 100)

https://github.com/Karlheinzniebuhr/pythonbenchmark

Granvillegranvillebarker

Answer 5

Have you reviewed the functionality provided profile or cProfile?

http://docs.python.org/library/profile.html

This provides much more detailed information than just printing the time before and after a function call. Maybe worth a look...

Barina

@CruiZen - of course, this question/answer was from 1.5 yrs ago! :) – Barina

Answer 6

The documentation also mentions that time.clock() and time.time() have different resolution depending on platform. On Unix, time.clock() measures CPU time as opposed to wall clock time.

timeit also disables garbage collection when running the tests, which is probably not what you want for production code.

I find that time.time() suffices for most purposes.

Etymology

Answer 7

From Python 2.6 on timeit is not limited to input string anymore. Citing the documentation:

Changed in version 2.6: The stmt and setup parameters can now also take objects that are callable without arguments. This will embed calls to them in a timer function that will then be executed by timeit(). Note that the timing overhead is a little larger in this case because of the extra function calls.

Catercorner