What exactly is contained within a obj.__closure__?

Question

Beazley pg 100 mentions:

>>>python.__closure__
(<cell at 0x67f50: str object at 0x69230>,)
>>>python.__closure__[0].cell_contents

my understanding is that __closure__ is a list but what's all this cell stuff and str object?? That looks like a 1-ary tuple?

Dory

Recently I've written an answer to the question of why we get "NameError: free variable 'var' referenced before assignment error. I explained your question too if you wanna take a look at: https://mcmap.net/q/57672/-how-can-quot-nameerror-free-variable-39-var-39-referenced-before-assignment-in-enclosing-scope-quot-occur-in-real-code – Toscano

Answer 1

Closure cells refer to values needed by the function but are taken from the surrounding scope.

When Python compiles a nested function, it notes any variables that it references but are only defined in a parent function (not globals) in the code objects for both the nested function and the parent scope. These are the co_freevars and co_cellvars attributes on the __code__ objects of these functions, respectively.

Then, when you actually create the nested function (which happens when the parent function is executed), those references are then used to attach a closure to the nested function.

A function closure holds a tuple of cells, one each for each free variable (named in co_freevars); cells are special references to local variables of a parent scope, that follow the values those local variables point to. This is best illustrated with an example:

def foo():
    def bar():
        print(spam)

    spam = 'ham'
    bar()
    spam = 'eggs'
    bar()
    return bar

b = foo()
b()

In the above example, the function bar has one closure cell, which points to spam in the function foo. The cell follows the value of spam. More importantly, once foo() completes and bar is returned, the cell continues to reference the value (the string eggs) even though the variable spam inside foo no longer exists.

Thus, the above code outputs:

>>> b=foo()
ham
eggs
>>> b()
eggs

and b.__closure__[0].cell_contents is 'eggs'.

Note that the closure is dereferenced when bar() is called; the closure doesn't capture the value here. That makes a difference when you produce nested functions (with lambda expressions or def statements) that reference the loop variable:

def foo():
    bar = []
    for spam in ('ham', 'eggs', 'salad'):
        bar.append(lambda: spam)
    return bar

for bar in foo():
    print bar()

The above will print salad three times in a row, because all three lambda functions reference the spam variable, not the value it was bound to when the function object was created. By the time the for loop finishes, spam was bound to 'salad', so all three closures will resolve to that value.

Enamelware

Just an FYI that the order of elements in __closure__ is arbitrary (hash table ordered). If you want to match the names of the free variables to their values, use a combination of __code__.co_freevars and __closure__[<index>].cell_contents. Index i co_freevars of the first corresponds to the index i of __closure__. – Wale

@IllyaGerasymchuk: or, just use the inspect.getclosurevars() function, which combines the names for you. – Enamelware

@IllyaGerasymchuk Your comment says index i (read: i) of the first corresponds to index i` (read: i prime) of the second. Is the prime a typo? Should it be code.co_freevars[i] corresponds to __closure__[i]? Also, the following link mentions __closure__, cell_content, and cell: docs.python.org/3/reference/… – Daggett

@joseville: the forward tick is a typo, it should be a backtick around i to mark that string as code. As a moderator I have the access to correct the comment, so I did. – Enamelware

Could you please explain what you mean by "the closure is dereferenced when bar() is called; the closure doesn't capture the value here." I have very little experience with dereferencing, and from I've just been reading online, it seems to me that "dereference" means to select the value held in memory at the address the reference points to, so your statement sounds contradictory to me. I'm hoping your elaboration would clear up my misunderstanding. – Weimer

Answer 2

It is the new Python 3 name for the old func_closure.

http://docs.python.org/3.0/whatsnew/3.0.html

The function attributes named func_X have been renamed to use the __X__ form, freeing up these names in the function attribute namespace for user-defined attributes. To wit, func_closure, func_code, func_defaults, func_dict, func_doc, func_globals, func_name were renamed to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__, __name__, respectively.

In a nutshell:

__closure__ is None or a tuple of cells that contain binding for the function's free variables.

Also, it is NOT writable.

Reference:https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy

Example Python < 3 (so I am using func_closure)

def foo():
    x = "I am used"
    y = "I am free"
    z = "I am free too"

    def bar(x):
        return x, y, z

    return bar

c = foo().func_closure

print [i.cell_contents for i in c]

Output:

>>> 
['I am free', 'I am free too']

As foo is returning the function bar which is using its own value x, but not y or z. So, they come under __closure__.

Dandy

The type.htlm link seems dead. Info about __closure__ and cell contents can be found here: docs.python.org/3/reference/… – Daggett

Answer 3

when a nested function(closure) is defined in python:
the outer function uses co_cellvars to note variables defined in the outer function that could be referenced by inner function.
the inner function uses co_freevars to note variables defined in the outer function, which could be referenced for later use.

Example:

# python3
Python 3.4.5 (default, May 29 2017, 15:17:55) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo(n):
...     a = 1
...     def g(n):
...             return a - n
...     return g
... 
>>> foo.__closure__
>>> foo.__code__.co_freevars
()
>>> foo.__code__.co_cellvars
('a',)
>>> foo(0).__closure__
(<cell at 0x7f2cd98db1c8: int object at 0x7f2cd9847960>,)
>>> foo(0).__closure__[0].cell_contents
1
>>> foo(0).__code__.co_freevars
('a',)
>>> foo(0).__code__.co_cellvars
()

Serg

Answer 4

>>> def f():
...     a = "HELO"
...     b = 1.0
...     def w(c):
...         return a,b,c
...     return w

>>> w = f()
>>> w.__closure__
(<cell at 0xa05c4ac: str object at 0x9e91b74>, <cell at 0xa05c3bc: float object at 0xb733dde8>)
>>> w.__closure__[0].cell_contents
'HELO'
>>> w.__closure__[1].cell_contents
1.0

I have never seen the cell type used anywhere else. It seems to be purpose-built to hold closure variables.

Kaylee