Why does locals() return a strange self referential list?

Question

So I'm using locals() to grab some arguments in the function. Works nicely:

def my_function(a, b):
    print locals().values()

>>> my_function(1,2)
[1, 2]

Standard stuff. But now let's introduce a list comprehension:

def my_function(a, b):
    print [x for x in locals().values()]

>>> my_function(1,2)
[[...], 1, 2]

Ehh? Why has it inserted a self-reference?

Melodiemelodion

Not in Python 2.7 it doesn't. Python 2.6, yes. – Kwangchowan

why do you only print the value? why don't you print the key as weel, to see what refers to what? – Korwun

@MartijnPieters was it a bug in 2.6? It disappears in 2.7 yes but unfortunately I don't have the luxury of a 2.7 installation. – Melodiemelodion

@LittleBobbyTables: most likely a change in the namespace handling of list comprehensions; the list comp output appears to be stored in the locals namespace. I see it in Python 2.5 as well. – Kwangchowan

@LittleBobbyTables: If you need to filter this out in a list comprehension use [v for k, v in locals().iteritems() if not k.startswith('_[')]. – Kwangchowan

Answer 1

Python versions before 2.7 and 3.1 used suboptimal bytecode to produce a list comprehension. In those Python versions, the list comprehension was stored in a local variable (or even a global, if at module scope):

>>> import dis
>>> def foo():
...     return [x for x in y]
... 
>>> dis.dis(foo)
  2           0 BUILD_LIST               0
              3 DUP_TOP             
              4 STORE_FAST               0 (_[1])
              7 LOAD_GLOBAL              0 (y)
             10 GET_ITER            
        >>   11 FOR_ITER                13 (to 27)
             14 STORE_FAST               1 (x)
             17 LOAD_FAST                0 (_[1])
             20 LOAD_FAST                1 (x)
             23 LIST_APPEND         
             24 JUMP_ABSOLUTE           11
        >>   27 DELETE_FAST              0 (_[1])
             30 RETURN_VALUE        

The _[1] local variable is the list-in-progress. When nesting list comprehensions it would use increasing integers to refer to the result:

>>> def bar():
...     return [[x for x in y] for z in spam]
... 
>>> dis.dis(bar)
  2           0 BUILD_LIST               0
              3 DUP_TOP             
              4 STORE_FAST               0 (_[1])
              7 LOAD_GLOBAL              0 (spam)
             10 GET_ITER            
        >>   11 FOR_ITER                40 (to 54)
             14 STORE_FAST               1 (z)
             17 LOAD_FAST                0 (_[1])
             20 BUILD_LIST               0
             23 DUP_TOP             
             24 STORE_FAST               2 (_[2])
             27 LOAD_GLOBAL              1 (y)
             30 GET_ITER            
        >>   31 FOR_ITER                13 (to 47)
             34 STORE_FAST               3 (x)
             37 LOAD_FAST                2 (_[2])
             40 LOAD_FAST                3 (x)
             43 LIST_APPEND         
             44 JUMP_ABSOLUTE           31
        >>   47 DELETE_FAST              2 (_[2])
             50 LIST_APPEND         
             51 JUMP_ABSOLUTE           11
        >>   54 DELETE_FAST              0 (_[1])
             57 RETURN_VALUE        

By looping over locals().values() you included a reference to the list-in-progress in the return value. Note that the bytecode uses a DELETE_FAST to clean up the local name to try and avoid the namespace pollution.

This was optimized for Python 3.1 and 2.7, see issue 2183. The list result under construction was moved to the stack instead. The optimization changed the LIST_APPEND bytecode to reference what list on the stack to append to, removing the need to use DUP_TOP -> STORE_FAST at the start, LOAD_FAST each iteration and DELETE_FAST after the list comprehension.

Kwangchowan