What is the scope of a defaulted parameter in Python? [duplicate]

Question

When you define a function in Python with an array parameter, what is the scope of that parameter?

This example is taken from the Python tutorial:

def f(a, L=[]):
    L.append(a)
    return L

print f(1)
print f(2)
print f(3)

Prints:

[1]
[1, 2]
[1, 2, 3]

I'm not quite sure if I understand what's happening here. Does this mean that the scope of the array is outside of the function? Why does the array remember its values from call to call? Coming from other languages, I would expect this behavior only if the variable was static. Otherwise it seems it should be reset each time. And actually, when I tried the following:

def f(a):
    L = []
    L.append(a)
    return L

I got the behavior I expected (the array was reset on each call).

So it seems to me that I just need the line def f(a, L=[]): explained - what is the scope of the L variable?

Lenient

See here: #1133441 – Sharleensharlene

@Sharleensharlene - thanks, the discussion there is great. This line strikes me as very descriptive and helpful - "[Python] binds the default argument at function definition, and not at function execution." – Lenient

Answer 1

The scope is as you would expect.

The perhaps surprising thing is that the default value is only calculated once and reused, so each time you call the function you get the same list, not a new list initialized to [].

The list is stored in f.__defaults__ (or f.func_defaults in Python 2.)

def f(a, L=[]):
    L.append(a)
    return L

print f(1)
print f(2)
print f(3)
print f.__defaults__
f.__defaults__ = (['foo'],) # Don't do this!
print f(4)

Result:

[1]
[1, 2]
[1, 2, 3]
([1, 2, 3],)
['foo', 4]

Sheepskin

But I don't really understand that. Where is that list stored? – Lenient

Thanks :). So variables declared within a function are only accessible within the scope of the function, but variables declared within a function signature / header / parameter list are put into a public scope and are available anywhere? – Lenient

@Charlie: Close, but not quite. The variable L still has local scope. When the function is called, L is set to the corresponding value in func_defaults, which in this case happens to be a reference to a mutable object. It is only the objects in func_defaults that you have access to from outside the function. It's best not to mess with them though. – Sheepskin

@Mark - so if I change the value in func_defaults, the function will use the updated value the next time it's called? (I'd understand that this isn't a good idea if so... why should Python even allow this? Does it?) – Lenient

@charlie: Yes you can in theory modify it, see my updated answer. But you shouldn't do this. It is evil and your colleagues will hate you forever if you do this. – Sheepskin

@Mark - :P I can understand that. Is there a reason why Python allows it? – Lenient

@charlie: Python lets you at a lot of its innards when 1) it doesn't hurt the language's performance, and 2) it allows cool meta-programming tricks. In this case, allowing runtime access to func_defaults allows a programmer to, for example, create a function that is a function factory, separating the specification of the function and its defaults. – Deary

For example, GetOpWithBoundDefault(operation, default) could take a string for operation, fetch a function object and list of allowed default argument types (and maybe other constraints) from a dict, then check that the type of default is compatible with said function before gluing them together and returning the new function object. I used to think that things like these were just too out there for real folks. But then I looked at the code behind Django's ORM, and learned about metaclasses, and after my head exploded, I stopped assuming that people wouldn't Go There. – Deary

This is crazy! Why doesn't python recalculate the default every time? And why do I get the feeling the answer to this question is actually very obvious... – Lerner

@Lerner Calculating the default could be something expensive, and not something you want to calculate every time. If you do want a dynamically chosen default, you can always use None (or some other suitable sentinel), and check for the sentinel in the body of the function at runtime. – Apraxia

Answer 2

The scope of the L variable is behaving as you expect.

The "problem" is with the list you're creating with []. Python does not create a new list each time you call the function. L gets assigned the same list each time you call which is why the function "remembers" previous calls.

So in effect this is what you have:

mylist = []
def f(a, L=mylist):
    L.append(a)
    return L

The Python Tutorial puts it this way:

The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes.

and suggests the following way to code the expected behaviour:

def f(a, L=None):
    if L is None:
        L = []
    L.append(a)
    return L

Binkley

The first code example you give is confusing because it suggests that the default parameter "L" has a global scope when in fact it only has scope within the function being defined. – Burlesque

@Burlesque - I'm not sure why you say that. To me the example is clearly saying that each time the function is called L gets assigned a value which is not recreated each time the function is called. – Binkley

Answer 3

There's even less "magic" than you might suspect. This is equivalent to

m = []

def f(a, L=m):
    L.append(a)
    return L

print f(1)
print f(2)
print f(3)

m is only created once.

Footbridge

Except, of course, that m's scope is not the same as L's in the original. – Lappet

This makes a bit more sense, as the list exists outside of the function. Thus it could make sense that it persists. But if it's defined within the function, it should be lost as soon as you exit. Does this mean that Python interprets the line "def f(a, L=[])" as the 2 lines "L = []" and then "def f(a, L)"? – Lenient

Of course. @Charlie: Its created at function definition time, not function execution time. Functions are also "just" objects (admittedly a bit "special", and with syntax sugar to create them), and as shown in the other comment their default parameters are stored in that object. – Footbridge

Answer 4

Say you have the following code:

def func(a=[]):
    a.append(1)
    print("A:", a)

func()
func()
func()

You can use python's indentation to help you understand what's going on. Everything that is flush to the left margin is executed when the file gets executed. Everything that's indented is compiled into a code object which gets executed when func() is called. So the function is defined and its default arguments set once, when the program gets executed (because the def statement is flush left).

What it does with the default arguments is an interesting issue. In python 3, it puts most of the information about a function in two places: func.__code__ and func.__defaults__. In python 2, func.__code__ was func.func_code func.__defaults__ was func.func_defaults. Later versions of python 2, including 2.6 have both sets of names, to aid the transition from python 2 to python 3. I will use the more modern __code__ and __defaults__. If you're stuck on an older python, the concepts are the same; just the names differ.

The default values are stored in func.__defaults__, and retrieved each time the function is called.

Thus when you define the function above, the body of the function gets compiled and stored in variables under __code__, to be executed later, and the default arguments get stored in __defaults__. When you call the function, it uses the values in __defaults__. If those values get modified for any reason, it only has the modified version available to use.

Play around defining different functions in the interactive interpreter, and see what you can figure out about how python creates and uses functions.

Giulietta

Answer 5

The explaination is given in answers to this question. To sum it up here:

Functions in Python are a kind of object. Because they are a kind of object, they act like objects when instantiated. A function, if defined with a mutable attribute as a default argument, is exactly the same as a class with a static attribute that is a mutable list.

Lennart Regebro has a good explanation and the answer to the question by Roberto Liffredo is excellent.

To adapt Lennart's answer ... if I have a BananaBunch class:

class BananaBunch:
    bananas = []

    def addBanana(self, banana):
        self.bananas.append(banana)


bunch = BananaBunch()
>>> bunch
<__main__.BananaBunch instance at 0x011A7FA8>
>>> bunch.addBanana(1)
>>> bunch.bananas
[1]
>>> for i in range(6):
    bunch.addBanana("Banana #" + i)
>>> for i in range(6):
    bunch.addBanana("Banana #" + str(i))

>>> bunch.bananas
[1, 'Banana #0', 'Banana #1', 'Banana #2', 'Banana #3', 'Banana #4', 'Banana #5']

// And for review ... 
//If I then add something to the BananaBunch class ...
>>> BananaBunch.bananas.append("A mutated banana")

//My own bunch is suddenly corrupted. :-)
>>> bunch.bananas
[1, 'Banana #0', 'Banana #1', 'Banana #2', 'Banana #3', 'Banana #4', 'Banana #5', 'A mutated banana']

How does this apply to functions? Functions in Python are objects. This bears repeating. Functions in Python are objects.

So when you create a function, you are creating an object. When you give a function a mutable default value, you are populating that object's attribute with a mutable value, and every time you call that function you are operating on the same attribute. So if you are using a mutable call (like append), then you are modifying the same object, just as if you were adding bananas to the bunch object.

Dresser

(Thanks for the SO link, it's great!) But then variables declared within the function should also be like attributes / member-level variables, and should also persist from call to call. No...? – Lenient

@charlie, not quite. The function object contains an attribute for default arguments, but not an attribute for other variables declared within it. See @Mark Byers answer for a more detailed explanation. – Dresser

Also, down voter, I cannot improve this post if I do not have feedback as to why I am wrong. ;-) – Dresser

Answer 6

The "problem" here is that L=[] is only evaluated once, that is, when the file is compiled. Python steps through each line of the file and compiles it. By the time it reaches the def with the default parameter, it creates that list instance once.

If you put L = [] inside the function code, the instance is not created at "compile time" (actually compile time can also be called part of the run time) because Python compiles the function's code but does not call it. So you will get a new list instance because the creation is done every time you call the function (instead of once during compilation).

A solution for that problem is not to use mutable objects as default parameters, or only fixed instances like None:

def f(a, L = None):
    if l == None:
        l = []
    L.append(a)
    return L

Note that in both cases you described, the scope of L is the function scope.

Achievement

Answer 7

You have to keep in mind that python is an interpreted language. What is happening here is when the function "f" is defined, it creates the list and assigns it to the default parameter "L" of function "f". Later, when you call this function, the same list is used as the default parameter. In short, the code on the "def" line, only gets executed once when the function is defined. This is a common python pitfall, of which I have fallen in myself.

def f(a, L=[]):
    L.append(a)
    return L

print f(1)
print f(2)
print f(3)

There have been suggestions for idioms in other answers here to fix this issue. The one I would suggest is as follows:

def f(a, L=None):
    L = L or []
    L.append(a)
    return L

This uses the or short circuit to either take the "L" that was passed, or create a new list.

The answer to your scope question is the "L" only has a scope within the function "f", but because the default parameters are only assigned once to a single list instead of every time you call the function it behaves as if the default parameter "L" has a global scope.

Burlesque

Note that you only have to use the idiom suggested for non-literals or in other words lists/dicts/classes – Burlesque

Abusing the short-circuiting behavior of or this way is less readable than using an if statement explicitly (if L is not None:). For example, if I passed in an empty instance of my own list-like type, I would hope f would use it, but it wouldn't, though it would with a non-empty instance. – Lithosphere

Ah, in the case of lists you are correct. I guess I've never been bitten with this because when I pass a list as a parameter I don't expect to use it after the function/method call (like a pointer in C). As for readability, I find it perfectly readable whereas if you had say 5 default parameters you would need 5 if statements filling most of the top half of your function which just looks ugly to me. This is all subjective of course. But thanks for you informative comment. – Burlesque