Scope of lambda functions and their parameters? [duplicate]

Question

I need a callback function that is almost exactly the same for a series of gui events. The function will behave slightly differently depending on which event has called it. Seems like a simple case to me, but I cannot figure out this weird behavior of lambda functions.

So I have the following simplified code below:

def callback(msg):
    print msg

#creating a list of function handles with an iterator
funcList=[]
for m in ('do', 're', 'mi'):
    funcList.append(lambda: callback(m))
for f in funcList:
    f()

#create one at a time
funcList=[]
funcList.append(lambda: callback('do'))
funcList.append(lambda: callback('re'))
funcList.append(lambda: callback('mi'))
for f in funcList:
    f()

The output of this code is:

mi
mi
mi
do
re
mi

I expected:

do
re
mi
do
re
mi

Why has using an iterator messed things up?

I've tried using a deepcopy:

import copy
funcList=[]
for m in ('do', 're', 'mi'):
    funcList.append(lambda: callback(copy.deepcopy(m)))
for f in funcList:
    f()

But this has the same problem.

Kuykendall

Your question title is somewhat misleading. – Romain

Why use lambdas if you find them confusing? Why not use def to define functions? What is it about your problem that makes lambdas so important? – Ticker

@Ticker Nested function will result in same problem (maybe more clearly visible) – Romain

@agartland: Are you me? I too was working on GUI events, and I wrote the following almost identical test before finding this page during background research: pastebin.com/M5jjHjFT – Tolbooth

See Why do lambdas defined in a loop with different values all return the same result? in the official Programming FAQ for Python. It explains the problem pretty nicely, and offers a solution. – Impressive

Related question, python - How do lexical closures work? - Stack Overflow – Perceptible

Answer 1

The problem here is the m variable (a reference) being taken from the surrounding scope. Only parameters are held in the lambda scope.

To solve this you have to create another scope for lambda:

def callback(msg):
    print msg

def callback_factory(m):
    return lambda: callback(m)

funcList=[]
for m in ('do', 're', 'mi'):
    funcList.append(callback_factory(m))
for f in funcList:
    f()

In the example above, lambda also uses the surounding scope to find m, but this time it's callback_factory scope which is created once per every callback_factory call.

Or with functools.partial:

from functools import partial

def callback(msg):
    print msg

funcList=[partial(callback, m) for m in ('do', 're', 'mi')]
for f in funcList:
    f()

Romain

This explanation is a bit misleading. The problem is the change of value of m in the iteration, not the scope. – Skippie

The comment above it true as noted by @Impressive in the comment on the question where a link is also given that explains the phenonimon and the solution. The factory method delivers the same effect as the argument to the factory method has the effect of creating a new variable with scope local to the lambda. However the solition given does not work syntatcically as there are no arguments to the lambda - and the lambda in lamda solution below also delivers the same effect without creating a new persistent method to create a lambda – Mollymollycoddle

Answer 2

When a lambda is created, it doesn't make a copy of the variables in the enclosing scope that it uses. It maintains a reference to the environment so that it can look up the value of the variable later. There is just one m. It gets assigned to every time through the loop. After the loop, the variable m has value 'mi'. So when you actually run the function you created later, it will look up the value of m in the environment that created it, which will by then have value 'mi'.

One common and idiomatic solution to this problem is to capture the value of m at the time that the lambda is created by using it as the default argument of an optional parameter. You usually use a parameter of the same name so you don't have to change the body of the code:

for m in ('do', 're', 'mi'):
    funcList.append(lambda m=m: callback(m))

Affinitive

i mean optional parameters with default values – Romain

Nice solution! Although tricky, I feel the original meaning is clearer than with other syntaxes. – Merras

There's nothing at all hackish or tricky about this; it's exactly the same solution the official Python FAQ suggests. See here. – Impressive

@abernert, "hackish and tricky" is not necessarily incompatible with "being the solution the official Python FAQ suggests". Thanks for the reference. – Erasmo

reusing the same variable name is unclear to someone unfamiliar with this concept. Illustration would be better if it were lambda n=m. Yes, you'd have to change your callback param, but the for loop body could stay the same i think. – Martinsen

+1 All the way! This should be the solution... The accepted solution is not nearly as good as this one. After all, we're trying to use lambda functions here... Not simply move everything to a def declaration. – Krissie

+1, this should definitely be the accepted solution, as not only does it avoid using a def (we're using lambda's here) but it also is the solution proposed in the python docs right under where it says "In order to avoid this, you need to save the values in variables local to the lambdas, so that they don’t rely on the value of the global variable" – All

@Affinitive can we say that Python lambda functions have dynamic scoping? – Apollyon

@aretor: No, it has lexical scoping. The m is from the lexically enclosing scope. – Affinitive

yoooo this answer is fire lol – Cwmbran

Answer 3

Python does uses references of course, but it does not matter in this context.

When you define a lambda (or a function, since this is the exact same behavior), it does not evaluate the lambda expression before runtime:

# defining that function is perfectly fine
def broken():
    print undefined_var

broken() # but calling it will raise a NameError

Even more surprising than your lambda example:

i = 'bar'
def foo():
    print i

foo() # bar

i = 'banana'

foo() # you would expect 'bar' here? well it prints 'banana'

In short, think dynamic: nothing is evaluated before interpretation, that's why your code uses the latest value of m.

When it looks for m in the lambda execution, m is taken from the topmost scope, which means that, as others pointed out; you can circumvent that problem by adding another scope:

def factory(x):
    return lambda: callback(x)

for m in ('do', 're', 'mi'):
    funcList.append(factory(m))

Here, when the lambda is called, it looks in the lambda' definition scope for a x. This x is a local variable defined in factory's body. Because of this, the value used on lambda execution will be the value that was passed as a parameter during the call to factory. And doremi!

As a note, I could have defined factory as factory(m) [replace x by m], the behavior is the same. I used a different name for clarity :)

You might find that Andrej Bauer got similar lambda problems. What's interesting on that blog is the comments, where you'll learn more about python closure :)

Swanherd

Answer 4

Yes, that's a problem of scope, it binds to the outer m, whether you are using a lambda or a local function. Instead, use a functor:

class Func1(object):
    def __init__(self, callback, message):
        self.callback = callback
        self.message = message
    def __call__(self):
        return self.callback(self.message)
funcList.append(Func1(callback, m))

Estep

Answer 5

the soluiton to lambda is more lambda

In [0]: funcs = [(lambda j: (lambda: j))(i) for i in ('do', 're', 'mi')]

In [1]: funcs
Out[1]: 
[<function __main__.<lambda>>,
 <function __main__.<lambda>>,
 <function __main__.<lambda>>]

In [2]: [f() for f in funcs]
Out[2]: ['do', 're', 'mi']

the outer lambda is used to bind the current value of i to j at the

each time the outer lambda is called it makes an instance of the inner lambda with j bound to the current value of i as i's value

Prosecute

Answer 6

First, what you are seeing is not a problem, and not related to call-by-reference or by-value.

The lambda syntax you defined has no parameters, and as such, the scope you are seeing with parameter m is external to the lambda function. This is why you are seeing these results.

Lambda syntax, in your example is not necessary, and you would rather be using a simple function call:

for m in ('do', 're', 'mi'):
    callback(m)

Again, you should be very precise about what lambda parameters you are using and where exactly their scope begins and ends.

As a side note, regarding parameter passing. Parameters in python are always references to objects. To quote Alex Martelli:

The terminology problem may be due to the fact that, in python, the value of a name is a reference to an object. So, you always pass the value (no implicit copying), and that value is always a reference. [...] Now if you want to coin a name for that, such as "by object reference", "by uncopied value", or whatever, be my guest. Trying to reuse terminology that is more generally applied to languages where "variables are boxes" to a language where "variables are post-it tags" is, IMHO, more likely to confuse than to help.

Gangrel

Answer 7

The variable m is being captured, so your lambda expression always sees its "current" value.

If you need to effectively capture the value at a moment in time, write a function takes the value you want as a parameter, and returns a lambda expression. At that point, the lambda will capture the parameter's value, which won't change when you call the function multiple times:

def callback(msg):
    print msg

def createCallback(msg):
    return lambda: callback(msg)

#creating a list of function handles with an iterator
funcList=[]
for m in ('do', 're', 'mi'):
    funcList.append(createCallback(m))
for f in funcList:
    f()

Output:

do
re
mi

Madlin

Answer 8

there are actually no variables in the classic sense in Python, just names that have been bound by references to the applicable object. Even functions are some sort of object in Python, and lambdas do not make an exception to the rule :)

Accordance

When you say "in the classic sense," you mean, "like C has." Plenty of languages, including Python, implement variables differently than C. – Kaylor

Answer 9

As a side note, map, although despised by some well known Python figure, forces a construction which prevents this pitfall.

fs = map (lambda i: lambda: callback (i), ['do', 're', 'mi'])

NB : the first lambda i acts like the factory in other answers.

Stultz