Python: is there a way to import a variable using timeit.timeit()?

Question

Suppose I have some function that takes an array and changes every element to be 0.

def function(array):
    for i in range(0,len(array)):
        array[i] = 0
return array

I want to test how long this function takes to run on a random array, which I wish to generate OUTSIDE of the timeit test. In other words, I don't want to include the time it takes to generate the array into the time.

I first store a random array in a variable x and do:

timeit.timeit("function(x)",setup="from __main__ import function")

But this gives me an error: NameError: global name 'x' is not defined

How can I do this?

Contempt

Possible duplicate of Getting "global name 'foo' is not defined" with Python's timeit – Subtilize

For Python 3.5+ check out the answer here – Langmuir

Answer 1

Import x from __main__ as well:

timeit.timeit("function(x)", setup="from __main__ import function, x")

Just like function, x is a name in the __main__ module, and can be imported into the timeit setup.

Kiosk

wow, that was silly of me not to think of that. Thank you, Martijn! – Contempt

How would I include multiple lines of setup? – Rosen

@cbracketdash: usually by executing your setup before you get to that point. I only ever the setup argument to set up the namespace, so import names from __main__. – Kiosk

Answer 2

You can avoid this problem entirely if you pass timeit a function instead of a string. In that case, the function executes in its normal globals and closure environment. So:

timeit.timeit(lambda: function(x))

Or, if you prefer:

timeit.timeit(partial(function, x))

(See here for details. Note that it requires Python 2.6+, so if you need 2.3-2.5, you can't use this trick.)

---

As the documentation says, "Note that the timing overhead is a little larger in this case because of the extra function calls."

This means that it makes timeit itself run slower. For example:

>>> def f(): pass
>>> timeit.timeit('timeit.timeit("f()", setup="from __main__ import f")', setup='import timeit', number=1000)
91.66315175301861
>>> timeit.timeit(lambda: timeit.timeit(f), number=100)
94.89793294097762

However, it doesn't affect the actual results:

>>> timeit.timeit(f, number=100000000)
8.81197881908156
>>> timeit.timeit('f()', setup='from __main__ import f', number=100000000)
8.893913001054898

(In the rare cases where it does, that typically means one version or the other wasn't testing the function the way it will be called in your real code, or was testing the wrong closure or similar.)

Note that the actual time taken inside the function here is about 88 seconds, so we've nearly doubled the overhead of timing code… but it's still only added 3% to the total testing time. And the less trivial f is, the smaller this difference will be.

Salesclerk

But note that you need to do this consistently across all tests so as not to let the additional cost of stack frame ops influence your comparisons. – Kiosk

@MartijnPieters: In normal cases, there are no extra stack frame ops inside the part being timed, only in the cost of timing itself. For example, with an empty function, I get 29.3ns per loop either way… but running timeit on the timeit itself, it's taking 94ms instead of 91ms to find that 29.3ns. – Salesclerk

(You might want to double-check my test: timeit.timeit(lambda: timeit.timeit(f), number=1000) is trivial; timeit.timeit('timeit.timeit("f()", setup="from __main__ import f")', setup='import timeit', number=1000) a bit less so…) – Salesclerk

Is the fact that you can pass timeit.timeit() a function instead of a string documented somewhere and if so please point me to it? Thanks. – Federative

@martineau: Yes, it's documented exactly where I linked in my answer. (The 3.x version is here, but it's basically the same.) – Salesclerk

What you linked to in your answer is the documentation for the timeit.Timer class not the timeit.timeit function. Regardless, neither look like they accept an argument that could be the function to be timed -- or am I missing something? – Federative

@martineau: Changed in version 2.6: The stmt and setup parameters can now also take objects that are callable without arguments. Functions are callable objects. timeit.timeit() is documented as a shorthand function for Timer() instances. – Kiosk

@Martijn: Ah, that must be the "trick" abarnert was alluding to -- guess I should try to pay bit more attention to the "Changed in a version < the ones I'm using" notes. Anyway, thanks! – Federative

@martineau: The timeit function says "Create a Timer instance with the given… and run its timeit method", which tells you to look at Timer for what the arguments mean, which is why I linked to Timer instead of timeit. (Also, note that it's only the 2.6 and 2.7 docs that say "Changed in version 2.6"; in the current docs it's just described as part of the functionality.) – Salesclerk

In your "doesn't affect the actual result" paragraph, you're not really showing that the function approach doesn't add overhead, since you're timing the function you created solely to wrap the thing you really want to time. Compare instead: timeit.timeit(lambda: None) and timeit.timeit("pass"). – Dutiable

Answer 3

Import x from __main__:

timeit.timeit("function(x)",setup="from __main__ import function, x")

Matrilineal

Answer 4

With Python 3.5, the optional argument globals has been introduced. It makes it possible to specify the namespace in which the timeit statement shall be executed.

So instead of writing:

timeit.timeit("function(x), setup="from __main__ import function, x")

...you can now write:

timeit.timeit("function(x)", globals=globals())

Langmuir

How would I include multiple lines of setup? – Rosen

@cbracketdash It depends on what you need. If your function only needs access to globals you can do the setup in regular code and use globals like shown. Additionally or if you can't just use globals you still can use the setup parameter like setup="print('line1'); print('line2')" where you use the semicolon to separate lines. – Langmuir

Answer 5

Alternatively, you can add x to globals. The good thing about it is that it works in a pdb debugging session:

globals()['x'] = x
timeit.timeit(lambda: function(x))

Note that the timing overhead is a little larger in this case because of the extra function calls. [source]

Zebra