How do lexical closures work? [duplicate]

Question

While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python:

flist = []

for i in xrange(3):
    def func(x): return x * i
    flist.append(func)

for f in flist:
    print f(2)

Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4".

This equivalent Perl code does it right:

my @flist = ();

foreach my $i (0 .. 2)
{
    push(@flist, sub {$i * $_[0]});
}

foreach my $f (@flist)
{
    print $f->(2), "\n";
}

"0 2 4" is printed.

Can you please explain the difference ?

---

Update:

The problem is not with i being global. This displays the same behavior:

flist = []

def outer():
    for i in xrange(3):
        def inner(x): return x * i
        flist.append(inner)

outer()
#~ print i   # commented because it causes an error

for f in flist:
    print f(2)

As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".

Sidereal

Here are two related questions that might help: What are Early and Late Binding? Why results of map() and list comprehension are different? – Rashida

Here is a pretty good article on this issue. me.veekun.com/blog/2011/04/24/gotcha-python-scoping-closures – Finespun

"Note that this example mindfully avoids lambda" Not relevant; named functions work the same way, for the same reasons. – Homecoming

Answer 1

Python is actually behaving as defined. Three separate functions are created, but they each have the closure of the environment they're defined in - in this case, the global environment (or the outer function's environment if the loop is placed inside another function). This is exactly the problem, though - in this environment, i is modified, and the closures all refer to the same i.

Here is the best solution I can come up with - create a function creater and invoke that instead. This will force different environments for each of the functions created, with a different i in each one.

flist = []

for i in xrange(3):
    def funcC(j):
        def func(x): return x * j
        return func
    flist.append(funcC(i))

for f in flist:
    print f(2)

This is what happens when you mix side effects and functional programming.

Argumentum

Your solution is also the one used in Javascript. – Sidereal

This is not misbehavior. It's behaving exactly as defined. – Clayton

you're right - i even say that myself at the end. i'll edit it to reflect this – Argumentum

It's the very same using lambda. – Aron

IMO piro has a better solution #234173 – Rashida

@J.F. Sebastian: you're right, looking back, I like that one more too =P. – Argumentum

I would perhaps change the innermost 'i' to 'j' for clarity. – Pestiferous

what about just defining it like this: def inner(x, i=i): return x * i – Ensample

funcC doesn't depend on i so you may as well hoist it out of the loop. – Globoid

Answer 2

The functions defined in the loop keep accessing the same variable i while its value changes. At the end of the loop, all the functions point to the same variable, which is holding the last value in the loop: the effect is what reported in the example.

In order to evaluate i and use its value, a common pattern is to set it as a parameter default: parameter defaults are evaluated when the def statement is executed, and thus the value of the loop variable is frozen.

The following works as expected:

flist = []

for i in xrange(3):
    def func(x, i=i): # the *value* of i is copied in func() environment
        return x * i
    flist.append(func)

for f in flist:
    print f(2)

Gascon

s/at compile time/at the moment when def statement is executed/ – Rashida

This is an ingenious solution, which makes it horrible. – Fionnula

There is one problem with this solution: func has now two parameters. That means it doesn't work with a variable amount of parameters. Worse, if you call func with a second parameter this will overwrite the original i from the definition. :-( – Stepparent

Answer 3

Here's how you do it using the functools library (which I'm not sure was available at the time the question was posed).

from functools import partial

flist = []

def func(i, x): return x * i

for i in range(3):
    flist.append(partial(func, i))

for f in flist:
    print(f(2))

Outputs 0 2 4, as expected.

Backwash

I really wanted to use this but my function is actully a class method and the first value passed is self. Is there anyway around this? – Cube

Absolutely. Suppose you have a class Math with a method add(self, a, b), and you want to set a=1 to create the 'increment' method. Then, create an instance of you class 'my_math', and your increment method will be 'increment = partial(my_math.add, 1)'. – Backwash

To apply this technique to a method you could also use functools.partialmethod() as of python 3.4 – Dagenham

'xrange' should be 'range'? – Gametophore

yes - the original code was written for python 2 (since it was 2011). Updated to python3. – Backwash

Answer 4

look at this:

for f in flist:
    print f.func_closure


(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)

It means they all point to the same i variable instance, which will have a value of 2 once the loop is over.

A readable solution:

for i in xrange(3):
        def ffunc(i):
            def func(x): return x * i
            return func
        flist.append(ffunc(i))

Husch

My question is more "general". Why has Python this flaw ? I would expect a language supporting lexical closures (like Perl and the whole Lisp dynasty) to work this out correctly. – Sidereal

Asking why something has a flaw is assuming it is not a flaw. – Husch

Answer 5

What is happening is that the variable i is captured, and the functions are returning the value it is bound to at the time it is called. In functional languages this kind of situation never arises, as i wouldn't be rebound. However with python, and also as you've seen with lisp, this is no longer true.

The difference with your scheme example is to do with the semantics of the do loop. Scheme is effectively creating a new i variable each time through the loop, rather than reusing an existing i binding as with the other languages. If you use a different variable created external to the loop and mutate it, you'll see the same behaviour in scheme. Try replacing your loop with:

(let ((ii 1)) (
  (do ((i 1 (+ 1 i)))
      ((>= i 4))
    (set! flist 
      (cons (lambda (x) (* ii x)) flist))
    (set! ii i))
))

Take a look here for some further discussion of this.

[Edit] Possibly a better way to describe it is to think of the do loop as a macro which performs the following steps:

1. Define a lambda taking a single parameter (i), with a body defined by the body of the loop,
2. An immediate call of that lambda with appropriate values of i as its parameter.

ie. the equivalent to the below python:

flist = []

def loop_body(i):      # extract body of the for loop to function
    def func(x): return x*i
    flist.append(func)

map(loop_body, xrange(3))  # for i in xrange(3): body

The i is no longer the one from the parent scope but a brand new variable in its own scope (ie. the parameter to the lambda) and so you get the behaviour you observe. Python doesn't have this implicit new scope, so the body of the for loop just shares the i variable.

Yolk

Interesting. I wasn't aware of the difference in semantics of the do loop. Thanks – Sidereal

Answer 6

The problem is that all of the local functions bind to the same environment and thus to the same i variable. The solution (workaround) is to create separate environments (stack frames) for each function (or lambda):

t = [ (lambda x: lambda y : x*y)(x) for x in range(5)]

>>> t[1](2)
2
>>> t[2](2)
4

Gaff

Answer 7

I'm still not entirely convinced why in some languages this works one way, and in some another way. In Common Lisp it's like Python:

(defvar *flist* '())

(dotimes (i 3 t)
  (setf *flist* 
    (cons (lambda (x) (* x i)) *flist*)))

(dolist (f *flist*)  
  (format t "~a~%" (funcall f 2)))

Prints "6 6 6" (note that here the list is from 1 to 3, and built in reverse"). While in Scheme it works like in Perl:

(define flist '())

(do ((i 1 (+ 1 i)))
    ((>= i 4))
  (set! flist 
    (cons (lambda (x) (* i x)) flist)))

(map 
  (lambda (f)
    (printf "~a~%" (f 2)))
  flist)

Prints "6 4 2"

And as I've mentioned already, Javascript is in the Python/CL camp. It appears there is an implementation decision here, which different languages approach in distinct ways. I would love to understand what is the decision, exactly.

Sidereal

The difference is in the (do ...) rather than the scoping rules. In scheme do creates a new variable every pass through the loop, while other languages reuse the existing binding. See my answer for more details and an example of a scheme version with similar behaviour to lisp/python. – Yolk

Answer 8

The variable i is a global, whose value is 2 at each time the function f is called.

I would be inclined to implement the behavior you're after as follows:

>>> class f:
...  def __init__(self, multiplier): self.multiplier = multiplier
...  def __call__(self, multiplicand): return self.multiplier*multiplicand
... 
>>> flist = [f(i) for i in range(3)]
>>> [g(2) for g in flist]
[0, 2, 4]

Response to your update: It's not the globalness of i per se which is causing this behavior, it's the fact that it's a variable from an enclosing scope which has a fixed value over the times when f is called. In your second example, the value of i is taken from the scope of the kkk function, and nothing is changing that when you call the functions on flist.

Clayton

Answer 9

The reasoning behind the behavior has already been explained, and multiple solutions have been posted, but I think this is the most pythonic (remember, everything in Python is an object!):

flist = []

for i in xrange(3):
    def func(x): return x * func.i
    func.i=i
    flist.append(func)

for f in flist:
    print f(2)

Claudiu's answer is pretty good, using a function generator, but piro's answer is a hack, to be honest, as it's making i into a "hidden" argument with a default value (it'll work fine, but it's not "pythonic").

Assibilate

I think it depends on your python version. Now I am more experienced and I would no longer suggest this way of doing it. Claudiu's is the proper way to make a closure in Python. – Assibilate

This won't work on either Python 2 or 3 (they both output "4 4 4"). The func in x * func.i will always refer to the last function defined. So even though each function individually has the correct number stuck to it, they all end up reading from the last one anyway. – Kilmarx

Answer 10

I didn't like how solutions above created wrappers in the loop. Note: python 3.xx

flist = []

def func(i):
    return lambda x: x * i

for i in range(3):
    flist.append(func(i))

for f in flist:
    print f(2)

Florist