Creating functions (or lambdas) in a loop (or comprehension)

Question

I'm trying to create functions inside of a loop:

functions = []

for i in range(3):
    def f():
        return i
    functions.append(f)

Alternatively, with lambda:

functions = []

for i in range(3):
    functions.append(lambda: i)

The problem is that all functions end up being the same. Instead of returning 0, 1, and 2, all three functions return 2:

print([f() for f in functions])

• Expected output: [0, 1, 2]
• Actual output: [2, 2, 2]

Why is this happening, and what should I do to get 3 different functions that output 0, 1, and 2 respectively?

Poock

as a reminder to myself: docs.python-guide.org/en/latest/writing/gotchas/… – Grinder

Note that the problem might not appear to occur using a generator, if you then iterate over the generator and call each function. This is because everything is lazily evaluated, and thus happens equally "late" as the binding. The iteration variable for the loop increments, the next function or lambda is immediately created, and then said function or lambda is immediately called - with the current iteration value. The same applies for generator expressions. See stackoverflow.com/questions/49633868 for an example. – Implore

Solution: replace lambda: i by lambda i=i: i. Your code is now for i in range(3): functions.append(lambda i=i: i). – Manvil

Basically "i is not local to the loop but a global variable". At Late Binding Closures gotchas we read Python’s closures are late binding. This means that the values of variables used in closures are looked up at the time the inner function is called. Though correct, the real problem is that Python has closures and mutable state and variables that are not declared (why??!!), leading to code becoming hard to read. PEP-3104 is interesting to read in that context. – Trilbie

Answer 1

You're running into a problem with late binding -- each function looks up i as late as possible (thus, when called after the end of the loop, i will be set to 2).

Easily fixed by forcing early binding: change def f(): to def f(i=i): like this:

def f(i=i):
    return i

Default values (the right-hand i in i=i is a default value for argument name i, which is the left-hand i in i=i) are looked up at def time, not at call time, so essentially they're a way to specifically looking for early binding.

If you're worried about f getting an extra argument (and thus potentially being called erroneously), there's a more sophisticated way which involved using a closure as a "function factory":

def make_f(i):
    def f():
        return i
    return f

and in your loop use f = make_f(i) instead of the def statement.

Perky

how do you know how to fix these things? – Caulescent

@Caulescent it's mostly just experience, most people have faced these things on their own at some point. – Radio

Can you explain why it is working please? (You save me on callback generated in loop, arguments was allways the lasts of the loop so thank you!) – Dinesen

Both examples would be slightly less confusing if the function parameter had a different name than the global variable ultimately provided as an argument. def f(i_local=i) return i_local and def make_f(i_local): def f(): return i_local; return f, for example. – Terefah

Answer 2

The Explanation

The issue here is that the value of i is not saved when the function f is created. Rather, f looks up the value of i when it is called.

If you think about it, this behavior makes perfect sense. In fact, it's the only reasonable way functions can work. Imagine you have a function that accesses a global variable, like this:

global_var = 'foo'

def my_function():
    print(global_var)

global_var = 'bar'
my_function()

When you read this code, you would - of course - expect it to print "bar", not "foo", because the value of global_var has changed after the function was declared. The same thing is happening in your own code: By the time you call f, the value of i has changed and been set to 2.

The Solution

There are actually many ways to solve this problem. Here are a few options:

• Force early binding of i by using it as a default argument

  Unlike closure variables (like i), default arguments are evaluated immediately when the function is defined:

  for i in range(3):
      def f(i=i):  # <- right here is the important bit
          return i
  
      functions.append(f)
  

  To give a little bit of insight into how/why this works: A function's default arguments are stored as an attribute of the function; thus the current value of i is snapshotted and saved.

  >>> i = 0
  >>> def f(i=i):
  ...     pass
  >>> f.__defaults__  # this is where the current value of i is stored
  (0,)
  >>> # assigning a new value to i has no effect on the function's default arguments
  >>> i = 5
  >>> f.__defaults__
  (0,)
  

• Use a function factory to capture the current value of i in a closure

  The root of your problem is that i is a variable that can change. We can work around this problem by creating another variable that is guaranteed to never change - and the easiest way to do this is a closure:

  def f_factory(i):
      def f():
          return i  # i is now a *local* variable of f_factory and can't ever change
      return f
  
  for i in range(3):           
      f = f_factory(i)
      functions.append(f)
  

• Use functools.partial to bind the current value of i to f

  functools.partial lets you attach arguments to an existing function. In a way, it too is a kind of function factory.

  import functools
  
  def f(i):
      return i
  
  for i in range(3):    
      f_with_i = functools.partial(f, i)  # important: use a different variable than "f"
      functions.append(f_with_i)
  

Caveat: These solutions only work if you assign a new value to the variable. If you modify the object stored in the variable, you'll experience the same problem again:

>>> i = []  # instead of an int, i is now a *mutable* object
>>> def f(i=i):
...     print('i =', i)
...
>>> i.append(5)  # instead of *assigning* a new value to i, we're *mutating* it
>>> f()
i = [5]

Notice how i still changed even though we turned it into a default argument! If your code mutates i, then you must bind a copy of i to your function, like so:

• def f(i=i.copy()):
• f = f_factory(i.copy())
• f_with_i = functools.partial(f, i.copy())

Ettie

The original code already uses closures - and the closures are only ever created behind the scenes by Python. It's just that each function created in the loop in the OP, generates its closure from the same local namespace; whereas each call to f_factory creates a new stack frame with new local variables, which each closure will use separately. We can still modify i *within f_factory after creating (but before returning) f. – Implore

Answer 3

For those coming to this question using lambda:

The solution is simply to replace lambda: i by lambda i=i: i.

functions = []
for i in range(3): 
    functions.append(lambda i=i: i)
print([f() for f in functions])
# [0, 1, 2]

Example use-case: How to have a lambda function evaluate a variable now (and not postponed)

Manvil

Answer 4

The problem is that i gets bound to the loop variable, which changes on every iteration. There are various solutions to this problem, with varying levels of readability. Of the 4 snippets below, the bottom 3 are all correct, but I think only the bottom one is the most readable:

# Original: wrong
f_list = [
    (lambda x: x + i) for i in range(10)
]
print([f(3) for f in f_list])

# Correct, but not so intuitive
f_list = [
    (lambda x, i=i: x + i) for i in range(10)
]
print([f(3) for f in f_list])

# More intuitive, but not so readable
f_list = [
    (lambda i: (lambda x: x + i))(i) for i in range(10)
]
print([f(3) for f in f_list])

# More readable
get_f = lambda i: (lambda x: x + i)
f_list = [
    get_f(i) for i in range(10)
]
print([f(3) for f in f_list])

Ember

Answer 5

Not an answer, but a note.

This phenomenon can also be observed outside of a loop, where it becomes quite clear what happens:

g = 'hello'

def foo():
    return lambda x : (g,x)

f = foo()

print(f('world')); g = 'goodbye'; print(f('earth'))

We (evidently?) get:

('hello', 'world')
('goodbye', 'earth')

Or one can look at the code in the question, but add printout by calling locals() and globals()

functions = []

print(f"Before loop: local i = {locals().get('i')}, global i = {globals().get('i')}")

for i in range(3):
    print(f"Inside loop: local i = {locals().get('i')}, global i = {globals().get('i')}")
    def f():
        print(f"Inside f(): local i = {locals().get('i')}, global i = {globals().get('i')}")
        return i
    functions.append(f)

print(f"After loop: local i = {locals().get('i')}, global i = {globals().get('i')}")

print([f() for f in functions])

We get the following, explaining a lot:

Before loop: local i = None, global i = None
Inside loop: local i = 0, global i = 0
Inside loop: local i = 1, global i = 1
Inside loop: local i = 2, global i = 2
After loop: local i = 2, global i = 2
Inside f(): local i = None, global i = 2
Inside f(): local i = None, global i = 2
Inside f(): local i = None, global i = 2
[2, 2, 2]

Ultimately it is down to the fact that the i of the loop in the question is not local to the loop but global to the program (and that languages with mutable state with closures bolted-one have awkward corners). Also, the 'problematic' decision to design a post-ALGOL60 ALGOL-style language where variables need not be properly declared doesn't help, i.e. it makes reading a Python program quite difficult (not to mention large-ish Python programs economically hazardous). Apparently PEP-3104 had an idea for a necessary improvement in 2006, but it wasn't taken up.

Bonus round: Java

As I was unsure what Java would do, here is same problem as it appears in Java. Note that the compiler needs to do extra effort to detect whether the variable the closure sees is mutable (non-final) and visible in an outside context or just local or final. The whole phenomenon is not just accidental.

package org.example;

import java.util.*;

public class Main {


    private static String mutableOuterContextString;

    private static void print(String str) {
        System.out.println("  '" + str + "'");
    }

    private static void printStringWithIdentity(String str) {
        System.out.println("  " + stringWithIdentity(str));
    }

    private static String stringWithIdentity(String str) {
        return "'" + str + "' at " + Objects.toIdentityString(str);
    }

    private final static List<String> numberStrings = Collections.unmodifiableList(Arrays.asList("one", "two", "three"));

    // Here, closures will use the 'reference to String' as given
    // by 'str' at 'closure build time'. Each has been given a specific 'str'.
    // At 'closure call time', the closures created will properly print
    // "one", "two", "three".
    // This corresponds to "function returning function" approach in Python.

    public static List<Runnable> one_two_three_as_expected_1() {
        final List<Runnable> funcs = new ArrayList<>();
        numberStrings.forEach(str -> funcs.add(
                () -> print(str)
        ));
        return funcs;
    }

    // This is the same code as above, just more explicit.

    public static List<Runnable> one_two_three_as_expected_2() {
        final List<Runnable> funcs = new ArrayList<>();
        for (final String str : numberStrings) {
            funcs.add(
                    () -> print(str)
            );
        }
        return funcs;
    }

    // This is the same code as above, just even more explicit.
    // The closure is in fact "just a class" created by the compiler.

    private static class RunnableX implements Runnable {

        private final String str;

        public RunnableX(final String str) {
            this.str = str;
        }

        @Override
        public void run() {
            print(str);
        }
    }

    public static List<Runnable> one_two_three_as_expected_3() {
        final List<Runnable> funcs = new ArrayList<>();
        for (final String str : numberStrings) {
            funcs.add(new RunnableX(str));
        }
        return funcs;
    }

    // As in Python, an interaction between "mutable state in an
    // outside context" and closures leads to surprises.
    //
    // Syntactically, there is not much difference between the
    // closure closing over a local/final variable (str) or a
    // mutable variable from the outside context (mutableOuterContextString)
    // but the compiler must create some different code indeed.

    public static List<Runnable> threethreethree_by_accessing_outside_context_1() {
        final List<Runnable> funcs = new ArrayList<>();
        for (final String str : numberStrings) {
            mutableOuterContextString = str;
            funcs.add(
                    () -> printStringWithIdentity(mutableOuterContextString)
            );
        }
        return funcs;
    }


    // This should be the same code as above, just more explicit.
    // The closure is in fact "just a class" created by the compiler.

    private static class RunnableY implements Runnable {

        @Override
        public void run() {
            printStringWithIdentity(mutableOuterContextString);
        }
    }

    public static List<Runnable> threethreethree_by_accessing_outside_context_2() {
        final List<Runnable> funcs = new ArrayList<>();
        for (final String str : numberStrings) {
            mutableOuterContextString = str;
            funcs.add(new RunnableY());
        }
        return funcs;
    }

    // If the try to reproduce the "three three three" effect with a
    // variable in the local context, we get something that will not compile:
    // "Variable used in lambda expression should be final or effectively final"
    // at "System.out.println(curString2)"

    /*
    public static List<Runnable> three_three_three_this_will_not_compile() {
        final List<Runnable> funcs = new ArrayList<>();
        String curString2;
        for (final String str : numberStrings) {
            curString2 = str;
            funcs.add(() -> print(curString2)); // <--- won't compile
        }
        return funcs;
    }
    */

    // Fixing it Python-style
    // Note that we do not even need to declare a local variable inside the build_..() method.
    // Directly using the variable "outerStr" that has been passed-in is good enough.
    // It is not important whether it has been declared "final" or not in the method declaration.

    public static Runnable build_closure_with_its_own_local_variable(final String outerStr) {
        System.out.println("  Creating closure with a local reference for " + stringWithIdentity(outerStr));
        return () -> printStringWithIdentity(outerStr);
    }

    public static List<Runnable> three_three_three_fixed() {
        final List<Runnable> funcs = new ArrayList<>();
        for (final String str : numberStrings) {
            mutableOuterContextString = str;
            funcs.add(build_closure_with_its_own_local_variable(mutableOuterContextString));
        }
        return funcs;
    }

    public static void main(String[] args) {
        System.out.println("Print 'one', 'two', 'three' as expected, take 1");
        one_two_three_as_expected_1().forEach(r -> r.run());
        System.out.println("Print 'one', 'two', 'three' as expected, take 2");
        one_two_three_as_expected_2().forEach(r -> r.run());
        System.out.println("Print 'one', 'two', 'three' as expected, take 3");
        one_two_three_as_expected_3().forEach(r -> r.run());
        System.out.println("Print 'three', 'three', 'three', unexpectedly, take 1");
        threethreethree_by_accessing_outside_context_1().forEach(r -> r.run());
        System.out.println("Print 'three', 'three', 'three', unexpectedly, take 2");
        threethreethree_by_accessing_outside_context_2().forEach(r -> r.run());
        System.out.println("Print 'one', 'two', 'three' again by creating a local variable");
        three_three_three_fixed().forEach(r -> r.run());
    }
}

Output

Print 'one', 'two', 'three' as expected, take 1
  'one'
  'two'
  'three'
Print 'one', 'two', 'three' as expected, take 2
  'one'
  'two'
  'three'
Print 'one', 'two', 'three' as expected, take 3
  'one'
  'two'
  'three'
Print 'three', 'three', 'three', unexpectedly, take 1
  'three' at java.lang.String@77459877
  'three' at java.lang.String@77459877
  'three' at java.lang.String@77459877
Print 'three', 'three', 'three', unexpectedly, take 2
  'three' at java.lang.String@77459877
  'three' at java.lang.String@77459877
  'three' at java.lang.String@77459877
Print 'one', 'two', 'three' again by creating a local variable
  Creating closure with a local reference for 'one' at java.lang.String@87aac27
  Creating closure with a local reference for 'two' at java.lang.String@6ce253f1
  Creating closure with a local reference for 'three' at java.lang.String@77459877
  'one' at java.lang.String@87aac27
  'two' at java.lang.String@6ce253f1
  'three' at java.lang.String@77459877

Trilbie

Answer 6

To add onto @Aran-Fey's excellent answer, in the second solution you might also wish to modify the variable inside your function which can be accomplished with the keyword nonlocal:

def f_factory(i):
    def f(offset):
      nonlocal i
      i += offset
      return i
    return f

for i in range(3):           
    f = f_factory(i)
    print(f(10))

Waechter

Why would you want to do that though? The i is local to f_factory() and should be used only for creating the closure? – Trilbie

Answer 7

You can try like this:

l=[]
for t in range(10):
    def up(y):
        print(y)
    l.append(up)
l[5]('printing in 5th function')

Gemination

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Shorthand

Answer 8

You have to save the each of the i value in a separate space in memory e.g.:

class StaticValue:
    val = None

    def __init__(self, value: int):
        StaticValue.val = value

    @staticmethod
    def get_lambda():
        return lambda x: x*StaticValue.val


class NotStaticValue:
    def __init__(self, value: int):
        self.val = value

    def get_lambda(self):
        return lambda x: x*self.val


if __name__ == '__main__':
    def foo():
        return [lambda x: x*i for i in range(4)]

    def bar():
        return [StaticValue(i).get_lambda() for i in range(4)]

    def foo_repaired():
        return [NotStaticValue(i).get_lambda() for i in range(4)]

    print([x(2) for x in foo()])
    print([x(2) for x in bar()])
    print([x(2) for x in foo_repaired()])

Result:
[6, 6, 6, 6]
[6, 6, 6, 6]
[0, 2, 4, 6]

Chameleon

Answer 9

just modify the last line for

functions.append(f())

Edit: This is because f is a function - python treats functions as first-class citizens and you can pass them around in variables to be called later on. So what your original code is doing is appending the function itself to the list, while what you want to do is append the results of the function to the list, which is what the line above achieves by calling the function.

Greasy

OP wants to create a list of functions. Not a list of numbers. – Propound