"except Foo as bar" causes "bar" to be removed from scope [duplicate]

Question

Given the following code:

msg = "test"
try:
    "a"[1]
except IndexError as msg:
    print("Error happened")
print(msg)

Can somebody explain why this causes the following output in Python 3?

Error happened
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(msg)
NameError: name 'msg' is not defined

Piglet

Answer 1

msg in the except clause is in the same scope as msg on the first line.

But in Python 3 we have this new behavior too:

When an exception has been assigned using as target, it is cleared at the end of the except clause. This is as if

except E as N:
    foo

was translated to

except E as N:
    try:
        foo
    finally:
        del N

This means the exception must be assigned to a different name to be able to refer to it after the except clause. Exceptions are cleared because with the traceback attached to them, they form a reference cycle with the stack frame, keeping all locals in that frame alive until the next garbage collection occurs.

so, you "overwrite msg" in the exception handler, and exiting the handler will delete the variable to clear the traceback reference cycle.

Pastelist

This answers specifically the question how and why the previous msg gets deleted after the except:, including reference to the docs, so IMHO it should be the accepted answer. – Corbett

Answer 2

Yes, as soon as the exception is raised and msg is assigned with the new exception object, the original object has no more reference and is therefore deleted. The new exception object is also deleted as soon as it leaves the except block.

You can verify it by overriding the __del__ method of the object and the exception assigned to msg:

class A:
    def __del__(self):
        print('object deleted')
class E(Exception):
    def __del__(self):
        print('exception deleted')
msg = A()
try:
    raise E()
except E as msg:
    print("Error happened")

This outputs:

object deleted
Error happened
exception deleted
NameError: name 'msg' is not defined

Linis

It's much simpler to use weakref.ref to demonstrate this; __del__ has confusing semantics which vary between runtimes and python versions. – Allegorize

@LightnessRacesinOrbit Yeah, this is definitely strange. I mean: if you want a new scope then create a new scope and don't kill the reference to the first msg. If you don't want a new scope, then why kill the exception reference? – Stanley

It's not wonky at all. By using except ... as msg you're binding msg to the exception, which removes all references to the original object. Conversely, if except ... as m was used, it wouldn't result in any issue since the original binding msg isn't modified in the except block. – Polygenesis

@Polygenesis As Giacomo explains, it’s definitely wonky. The way Python works, it rebinds msg in the existing scope. But the scope of msg exists beyond the try block (and would continue to, if no exception were raised). The fact that a runtime condition determines the scope of a variable (which should be static) is a crass subversion of the type system. Since Python is dynamically typed this of course works but it’s nevertheless weird. What’s more, Python 2 handles this completely differently, and as expected. – Erlene

@Polygenesis It's wonky in the sense that it is unintuitive and does not match other popular languages (though whether this is itself "bad" is another debate) or even the rest of the same language – Stambaugh

@Polygenesis except that's not how scoping works. There is a reference to the original object, it's a couple lines later (remember, scope is extent in code-space, not extent in time). The problem seems to be that Python is failing to create a scope. – Millepede

@Millepede try-catch does not introduce a new scope, so no, after the exception block is run, there is no longer a reference to the original object that msg was bound to, since msg was rebound and then deleted by the except block. As @Konrad Rudolph explains, it's the way Python works. It's the same idea as x = A()\ x = B()\ del x\ print x. The behavior in Python2 is different than this, but still "unexpected." Instead of throwing a NameError, you simply get string index out of range. The difference between Py2 and Py3 is that in Py2 the exception is not deleted (see Uku Loskit's answer). – Polygenesis

@LightnessRacesinOrbit Agreed about the "bad" debate. I suppose it's not counter-intuitive to me since I primarily work in Python, and make it a habit to not create any bindings in an except block that have the same name as any other bindings. I don't find it counter-intuitive that that try-catch doesn't introduce new scope, though. I think that's simply what one is used to. – Polygenesis

This is a good reason to be semantic about using the term "binding" instead of "variable" when describing behavior in Python that would be unexpected in other languages. – Polygenesis

@Polygenesis "it's the way Python works": that is surely true. Nobody is denying that. But it doesn't mean that people shouldn't consider Python to be wonky. Scope rules in "other popular languages" allow programmers using those languages to avoid the mental burden of keeping track of the names they've used before they got to the except block (or analogous) so as to be able to avoid using existing names in the block; the interpreter or compiler does it for them. – Celeski

@Polygenesis What is the difference between msg=1;try:... except E as msg: and x=1;for x in something:...? At the end of the for x keeps the binding to the last iteration, I don't see any good reason to add a hidden del msg at the end of the except block. Python never worked in this way, all other statements do not work in this way, previous versions of python did not work in this case. As I said, from a "lexical-scope point of view" either you introduce a new scope (and msg keeps the old value) or you just rebind the name. Deleting a binding in that way shouldn't be possible – Dalrymple

@Dalrymple The difference is that in the context of an exception, after it has been handled, it was decided in Py3 to delete the exception. Py2 still has "unexpected" behavior with this same code, where after the except block is through, print msg will return string index out of range instead of test like you'd expect if you had traditional language scoping, so I don't agree with saying that "Python never worked this way." Again, the original object "test" which was bound to msg is deleted as soon as the exception is thrown at runtime, regardless of Python version. – Polygenesis

@Dalrymple In the case of a for loop, you still aren't in a different scope. In Python, if you create a binding inside of a for loop, then try printing it outside of the for loop, you won't have any error. Try that in "other popular languages" and you'll get an undefined variable error. It's a nuance of Python's fundamental binding/object structure. Perhaps that unsafe runtime dynamic behavior is wonky when compared to other languages, but it can be useful sometimes. Python doesn't scope like other languages, simple as that. (This should be in chat, but the auto-move link hasn't appeared...) – Polygenesis

@Polygenesis The python2 behaviour is what I expect. If except does not create a new scope the old msg should be gone after reassigning it to the exception, but there is no reason to delete the exception. We could easily do the same with for loop: add a hidden del <vars> at the end of the for (or the else block attached to it). Same with with: with something as x: ...; print(x). They decided to add this hidden del for no reason breaking all the conventions python already had. You are justifying this simply saying "because it's written in the standard", but ... – Dalrymple

@Polygenesis you did not give any rationale as why that decision was taken, given that it surely breaks the least astonishment rule. Maybe there was a reason why is not a good idea to have access to the exception object outside the except block, but nothing you have written gives any hint as to why that should be the case. – Dalrymple

The reason msg is deleted at the end of the except block has nothing to do with scope. They wanted to clear a reference cycle created by the addition of the __traceback__ attribute in Python 3. See PEP 3110. It's definitely confusing, and there's a strong argument to be made that Python should stop trying to support running with the GC off, but it's not a scope issue. – Chagrin

@Dalrymple And it seems others would expect that msg should still be "text" like it would in other languages (due to scope). I personally think that is much more intuitive than it holding the exception as in Py2, coming from other languages where there actually is a scope change when handling exceptions. My whole point here is that there is no scope like in a traditional language; I'm not defending anything about Python 3's rationale about deleting exceptions after processing. All I'm saying is that the "scope-less" behavior is a nuance of Python bindings, nothing more. – Polygenesis

@Chagrin That's good information. AFAICT they could have also decided to use a more complex translation that checked if the as target already existed and, if so, stored the reference in a new binding and after the "del msg" they could restore the old value. This would be the right way of doing it if keeping the exception around is an issue. They decided to implement just a rough adaptation to fix the issue which produces this wonky behaviour. Sure my proposal would too be a deviation from the "pureness" of scoping rules but I fell it's much less astonishing than the current situation – Dalrymple

@Chagrin Am I reading that right? Python changed the semantics of the language in a nonsensical way to work around a buggy GC? If so, good grief – Charteris

But why the message is "Name msg not defined", while it's just a referenced object what became inaccessible? Name msg is still defined tho, isn't it? – Uticas

@Maciek: Python only distinguishes between unbound and nonexistent variables for function locals. At global scope (which the example's code runs in), a deleted name and a name that was never defined are treated the same. – Chagrin

Answer 3

Exception blocks delete the caught variable at the end of the block, but they do not have their own scopes. So the sequence of events goes:

1) msg is set to some string in the local scope

2) msg is set to an IndexError object in the same local scope as 1

3) msg is deleted from the local scope when the Exception block ends

4) msg is no longer defined in the local scope, so the attempt to access it fails

Musteline