Why does IDLE 3.4 take so long on this program?

Question

EDIT: I'm redoing the question entirely. The issue has nothing to do with time.time()

Here's a program:

import time
start=time.time()
a=9<<(1<<26)             # The line that makes it take a while
print(time.time()-start)

This program, when saved as a file and run with IDLE in Python 3.4, takes about 10 seconds, even though 0.0 is printed out from time.time(). The issue is very clearly with IDLE, because when run from the command line this program takes almost no time at all.

Another program that has the same effect, as found by senshin, is:

def f(): 
    a = 9<<(1<<26)

I have confirmed that this same program, when run in Python 2.7 IDLE or from the command line on python 2.7 or 3.4, is near instantaneous.

So what is Python 3.4 IDLE doing that makes it take so long? I understand that calculating this number and saving it to memory is disk intensive, but what I'd like to know is why Python 3.4 IDLE performs this computation and write when Python 2.7 IDLE and command line Python presumably do not.

Lye

It doesn't take 10 seconds for me in fact it works as expected for me – Lynlyncean

@Lye why do you think the program takes around 10 seconds? A simple program like this should take almost no time to run -- for example, on my 2.7GHz mac mini, "time python isaacg.py" reports that it takes about 0.028s to run. – Photovoltaic

Have you tried running this from the terminal? Did it still have the same result? – Herzegovina

You might want to run that multiple times and take the average. – Gift

I think the issue might have something to do with IDLE, and not with python or time.time(). Is this reproducible for any of you on IDLE? – Lye

@Lye This is bizarre, but I can reproduce it in IDLE (with Python 3.4.0rc1 on Windows 7). Works as fast as you would expect in the terminal, though. – Prerequisite

This takes less than 0.009 seconds for me (IDLE, Python 2.7, win 7). – Twirl

Try this by itself in a file and run from IDLE: def f(): a = 9<<(1<<26). It makes IDLE lag for me (even without calling the function); does it do that for you? If so, it's probably related to IDLE pre-computing constants or something weird like that. – Prerequisite

@Prerequisite I get the same lag. Well, that answers my question. I'm going to edit this question to ask about why IDLE does this, though. – Lye

It's back to instantaneous if you do def f(): x = 9; a = x<<(1<<26) or the same thing with a variable replacing the 1 or the 26 instead. Further evidence in favor of IDLE precomputing constants or something. – Prerequisite

Answer 1

I would look at that line and pick it apart. You have:

9 << (1 << 26)

(1 << 26) is the first expression evaluated, and it produces a really large number. What this line is saying, is that you are going to multiply the number 1 by 2 to the power of 26, effectively producing the number 2 ** 26 in memory. This is not the problem however. You then shift 9 left by the count of 2 ** 26. This produces a number that is around 50 million digits long in memory (I cant even calculate it exactly!), because the shift left is just too big. Be careful in the future, as shifts by what seems to be small amounts do in fact grow very fast. If it was any larger, your program may have not run at all. Your expression mathematically evaluates to 9 * 2 ** (2 ** 26), if you were curious.

The ambiguity in the comment section is probably actually dealing with how this huge portion of memory is handled by python under the hood, and not IDLE.

EDIT 1:

I thing that what is happening, is that a mathematical expression evaluates to its answer, even when placed inside of a function that isn't called yet, only if the expression is self sufficient. This means that if a variable is used in the equation, the equation will be untouched in the byte code, and not evaluated until hard execution. The function has to be interpreted, and in that process, I think that your value is actually computed, resulting in the slower times. I am not sure about this, but I strongly suspect this behavior to be the root cause. Even if it is not so, you got to admit that 9<<(1<<26) kicks the computer in the behind, there's not much optimization that can be done there.

In[73]: def create_number():
            return 9<<(1<<26)
In[74]: #Note that this seems instantaneous, but try calling the function!
In[75]: %timeit create_number()
#Python environment crashes because task is too hard

There is a slight deception in this kind of testing however. When trying this with the regular timeit, I got:

In[3]: from timeit import timeit
In[4]: timeit(setup = 'from __main__ import create_number', stmt = 'create_number()', number = 1)
Out[4]: .004942887388800443

Also keep in mind that printing the value is not do-able, so something like:

In[102]: 9<<(1<<26)

should not even be attempted.

For even more added support:

I felt like a rebel, so I decided to see what would happen if I timeit the raw execution of the equation:

In[107]: %timeit 9<<(1<<26)
10000000 loops, best of 3: 22.8 ns per loop

In[108]: def empty(): pass
In[109]: %timeit empty()
10000000 loops, best of 3: 96.3 ns per loop

This is really fishy, because apparently this calculation happens faster than the time it takes Python to call an empty function, which is obviously not the case. I repeat, this is not instantaneous, but probably has something to do with retrieving an already calculated object somewhere in memory, and reusing that value to calculate the expression. Anyways, nice question.

Implacable

Sorry, but I don't think that's a complete answer. It doesn't explain why this observed time delay occurs when I use Python 3.4 IDLE, but not Python 2.7 IDLE or the command line. I'll reword my question to make that more clear. – Lye

What does the expression put into the command line look like? Did you even consider the fact that undercover memory reference issues can be taking place for different environments? – Implacable

Answer 2

I am really puzzled. Here are more results with 3.4.1. Running either of the first two lines from the editor in either 3.4.1 or 3.3.5 gives the same contrast.

>>> a = 1 << 26; b = 9 << a  # fast, , .5 sec
>>> c = 9 << (1 << 26)  # slow, about 3 sec
>>> b == c  # fast
True
>>> exec('d=9<<(1<<26)', globals())  # fast
>>> c == d  # fast
True

The difference between normal execution and Idle's is that Idle exec's code in an exec call like the above, except that the 'globals' passed to exec is not globals() but a dict configured to look like globals(). I do not know of any 2.7 -- 3.4 difference in Idle in this respect except for the change of exec from statement to function. How can exec'ing an exec be faster than a single exec? How can adding an intermediate binding be faster?

Iroquois