Python class scoping rules

Question

EDIT: Looks like this is a very old "bug" or, actually, feature. See, e.g., this mail

I am trying to understand the Python scoping rules. More precisely, I thought that I understand them but then I found this code here:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    class C:
        print(x)
        print(y)
        y = 1
func()

In Python 3.4 the output is:

xlocal
ytop

If I replace the inner class by a function then it reasonably gives UnboundLocalError. Could you explain me why it behaves this strange way with classes and what is the reason for such choice of scoping rules?

Overtask

It wont give any error unless you call the inner function – Selah

Interesting; y should be an unbound exception in the case of y, but it isn't even when you make C a global. Corner cases FTW! Class bodies are special; they do not create a new scope of their own. As such y = 1 would make it a local, but until that point it is a global in class statements only. – Villagomez

9.2. Python Scopes and Namespaces and 4.1. Naming and binding are a start, you might need to re-read them often till it sinks in. – Throng

if it's actual python3 code, then it should be print(x) and print(y). print x is python2 code. – Yakka

thanks, @Yakka . changed that. Actually it behaves same way in Python2 – Overtask

This question could be helpful #4297177 – Kuhn

Thanks @Kasra . It is strange however that the example 3 in that question works only in python3.4 and does not work in python3.1 – Overtask

dis.dis(func) with and without the y=1 assignment in C shows that the former only loads a closure for x and the latter loads a closure for x and y when C gets built. Thought that was interesting though it doesn't point to any rule. – Throng

Sounds like this is a bug and should be reported as such. – Chorography

@Kasra: that is the Python 2.1 What's New document. That is ancient history, not a recent change. – Villagomez

See also: #9506479 – Chuckle

Answer 1

TL;DR: This behaviour has existed since Python 2.1 PEP 227: Nested Scopes, and was known back then. If a name is assigned to within a class body (like y), then it is assumed to be a local/global variable; if it is not assigned to (x), then it also can potentially point to a closure cell. The lexical variables do not show up as local/global names to the class body.

---

On Python 3.4, dis.dis(func) shows the following:

>>> dis.dis(func)
  4           0 LOAD_CONST               1 ('xlocal')
              3 STORE_DEREF              0 (x)

  5           6 LOAD_CONST               2 ('ylocal')
              9 STORE_FAST               0 (y)

  6          12 LOAD_BUILD_CLASS
             13 LOAD_CLOSURE             0 (x)
             16 BUILD_TUPLE              1
             19 LOAD_CONST               3 (<code object C at 0x7f083c9bbf60, file "test.py", line 6>)
             22 LOAD_CONST               4 ('C')
             25 MAKE_CLOSURE             0
             28 LOAD_CONST               4 ('C')
             31 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             34 STORE_FAST               1 (C)
             37 LOAD_CONST               0 (None)
             40 RETURN_VALUE

The LOAD_BUILD_CLASS loads the builtins.__build_class__ on the stack; this is called with arguments __build_class__(func, name); where func is the class body, and name is 'C'. The class body is the constant #3 for the function func:

>>> dis.dis(func.__code__.co_consts[3])
  6           0 LOAD_NAME                0 (__name__)
              3 STORE_NAME               1 (__module__)
              6 LOAD_CONST               0 ('func.<locals>.C')
              9 STORE_NAME               2 (__qualname__)

  7          12 LOAD_NAME                3 (print)
             15 LOAD_CLASSDEREF          0 (x)
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             21 POP_TOP

  8          22 LOAD_NAME                3 (print)
             25 LOAD_NAME                4 (y)
             28 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             31 POP_TOP

  9          32 LOAD_CONST               1 (1)
             35 STORE_NAME               4 (y)
             38 LOAD_CONST               2 (None)
             41 RETURN_VALUE

Within the class body, x is accessed with LOAD_CLASSDEREF (15) while y is load with LOAD_NAME (25). The LOAD_CLASSDEREF is a Python 3.4+ opcode for loading values from closure cells specifically within class bodies (in previous versions, the generic LOAD_DEREF was used); the LOAD_NAME is for loading values from locals and then globals. However closure cells show up neither as locals nor globals.

Now, because the name y is stored to within the class body (35), it is consistently being used as not a closure cell but a local/global name. The closure cells do not show up as local variables to the class body.

This behaviour has been true ever since implementing PEP 227 - nested scopes. And back then BDFL stated that this should not be fixed - and thus it has been for these 13+ years.

---

The only change since PEP 227 is the addition of nonlocal in Python 3; if one uses it within the class body, the class body can set the values of the cells within the containing scope:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    class C:
        nonlocal y  # y here now refers to the outer variable
        print(x)
        print(y)
        y = 1

    print(y)
    print(C.y)

func()

The output now is

xlocal
ylocal
1
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    func()
  File "test.py", line 13, in func
    print(C.y)
AttributeError: type object 'C' has no attribute 'y'

That is, print(y) read the value of the cell y of the containing scope, and y = 1 set the value in that cell; in this case, no attribute was created for the class C.

Supper

It seems that I understood your explanation. Since y is stored it "bypasses" the usual mechanism. But doesn't it look like a bug to you? – Overtask

If Guido says it is not a bug, then it is a feature. ;) – Blaspheme

@ivanl: it is a known 'feature' of the implementation; see the email Antti links to. Class scopes are not like function scopes. – Villagomez

Answer 2

First focus on the case of a closure -- a function within a function:

x = "xtop"
y = "ytop"
def func():
    x = "xlocal"
    y = "ylocal"
    def inner():
 #       global y
        print(x)
        print(y)
        y='inner y'
        print(y)
    inner()  

Note the commented out global in inner If you run this, it replicates the UnboundLocalError you got. Why?

Run dis.dis on it:

>>> import dis
>>> dis.dis(func)
  6           0 LOAD_CONST               1 ('xlocal')
              3 STORE_DEREF              0 (x)

  7           6 LOAD_CONST               2 ('ylocal')
              9 STORE_FAST               0 (y)

  8          12 LOAD_CLOSURE             0 (x)
             15 BUILD_TUPLE              1
             18 LOAD_CONST               3 (<code object inner at 0x101500270, file "Untitled 3.py", line 8>)
             21 LOAD_CONST               4 ('func.<locals>.inner')
             24 MAKE_CLOSURE             0
             27 STORE_FAST               1 (inner)

 14          30 LOAD_FAST                1 (inner)
             33 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             36 POP_TOP
             37 LOAD_CONST               0 (None)
             40 RETURN_VALUE

Note the different access mode of x vs y inside of func. The use of y='inner y' inside of inner has created the UnboundLocalError

Now uncomment global y inside of inner. Now you have unambiguously create y to be the top global version until resigned as y='inner y'

With global uncommented, prints:

xlocal
ytop
inner y

You can get a more sensible result with:

x = "xtop"
y = "ytop"
def func():
    global y, x
    print(x,y)
    x = "xlocal"
    y = "ylocal"
    def inner():
        global y
        print(x,y)
        y = 'inner y'
        print(x,y)
    inner()    

Prints:

xtop ytop
xlocal ylocal
xlocal inner y

---

The analysis of the closure class is complicated by instance vs class variables and what / when a naked class (with no instance) is being executed.

The bottom line is the same: If you reference a name outside the local namespace and then assign to the same name locally you get a surprising result.

The 'fix' is the same: use the global keyword:

x = "xtop"
y = "ytop"
def func():
    global x, y
    x = "xlocal"
    y = "ylocal"
    class Inner:
        print(x, y)
        y = 'Inner_y'
        print(x, y) 

Prints:

xlocal ylocal
xlocal Inner_y

You can read more about Python 3 scope rules in PEP 3104

Suazo

I understand the behavior for functions you pointed out. But it is still not clear what is the reason for classes to behave differenly. – Overtask

Classes will be similar but you need to further qualify instance vs class variables and nested class and nested instance variables. Bottom line is the same: The presence of the assignment of y=1 make the reference print(y) above the assignment ambiguous inside the closure. – Suazo

I would understand if my code would fail with UnboundLocal , but it looks really confusing how y can "bypass" y_local. Not failing with UnboundLocal really looks like a bug – Overtask

There is a deference in how the interpreter treats each case, true. It is unusual to just have a class definition and execute code by its presence though. – Suazo

PEP 3104 only introduces the nonlocal statement and doesn't actually explain the behaviour. The behaviour is a long-standing PEP 227 side-effect, 3104 never altered that. – Villagomez

@MartijnPieters: I did not claim 3104 was definitive -- only the most recent. It references PEP 227 and has a general overview. – Suazo