Why doesn't __getattr__ work with __exit__?

Question

I came across this as a bit of a surprise while trying to work out another question.

This seemed extremely odd to me, I thought it was worth asking the question. Why doesn't __getattr__ appear to work with with?

if I make this object:

class FileHolder(object):
    def __init__(self,*args,**kwargs):
        self.f= file(*args,**kwargs)

    def __getattr__(self,item):
        return getattr(self.f,item)

and using it with with,

>>> a= FileHolder("a","w")
>>> a.write
<built-in method write of file object at 0x018D75F8>
>>> with a as f:
...   print f
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: __exit__
>>> a.__exit__
<built-in method __exit__ of file object at 0x018D75F8>

Why does this happen?

EDIT

>>> object.__exit__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'object' has no attribute '__exit__'

It definitely isn't inheriting __exit__

Confutation

Something is going on here, in your class definition you make FileHolder a subclass of object. But in your code below it, it says that a is a file object. Thats not consistent. – Tell

@Tell Honestly it isn't. Test it yourself :) – Confutation

@jedwards, the __exit__ is from the file object assigned to self.f, if you ask type(a), you'll get FileHolder. – Superficies

@Adam, you're right -- I actually didn't create the class (I did something like class FileHolder(object): pass) -- serves me right. – Tell

Answer 1

I can't say for sure, but after reading over the PEP describing the with statement:

http://www.python.org/dev/peps/pep-0343/

This jumped out at me:

A new statement is proposed with the syntax:

    with EXPR as VAR:
        BLOCK

....

The translation of the above statement is:

    mgr = (EXPR)
    exit = type(mgr).__exit__  # Not calling it yet
    value = type(mgr).__enter__(mgr)

....

Right there. The with statement does not call __getattr__(__exit__) but calls type(a).__exit__ which does not exist giving the error.

So you just need to define those:

class FileHolder(object):                                                                                                                 
    def __init__(self,*args,**kwargs):
        self.f= file(*args,**kwargs)

    def __enter__(self,*args,**kwargs):
        return self.f.__enter__(*args,**kwargs)

    def __exit__(self,*args,**kwargs):
        self.f.__exit__(*args,**kwargs)

    def __getattr__(self,item):
        return getattr(self.f,item)

Eolithic

Surprisingly, the documentation is not-quite-right on this one -- special method lookup also ignores type(x).__getattr__. See my answer... – Inurn

I can't say for sure. My only guess is it's that way by design. It might be something to ask on the mailing list. – Eolithic

@GP89: I did some more digging; it's by design. See here for the full reasoning. – Inurn

I also did some digging in the mailing list @Inurn mention. Here is a short summary: mail.python.org/pipermail/python-dev/2009-May/089576.html – Eolithic

Answer 2

The with statement opcode SETUP_WITH looks up __exit__ as a "special method lookup", which ignores __getattr__ and __getattribute__ on new-style classes (but not on old-style classes). See this mailing list thread for more information, where they discuss adding the special method lookup semantics to with (which they eventually do). See also special method lookup for new-style classes for a detailed discussion on why these special methods are looked up in this way.

In particular, special method lookup also bypasses __getattr__ on the type object. So, even though the documentation says the method is looked up as type(mgr).__exit__, this code doesn't work:

class M(type):
    def __getattr__(*args): return lambda: 0

class X(object):
    __metaclass__ = M

x = X()
type(x).__exit__ # works, returns a lambda

with x: pass # fails, AttributeError

Inurn

It does beg the question though, why does __enter__ not get an AttributeError? unless it looks for exit first which seems unlikely – Confutation

As @korylprince's answer indicates, __exit__ is looked up first, so that they don't run into issues with calling __enter__ without a functional __exit__. You can see the exact CPython implementation here. – Inurn

Answer 3

The previous answers has explained the fact that __getattr__ does not work with __enter__ and __exit__. I'm here to give my thinking of why it SHOULD NOT work.

The only reason we define __enter__ and __exit__ methods on an object is that we need to use it in with statement. The two methods help us get and release a resource implicitly, so we usually define them like this:

class Resource(object):
    ...
    def __enter__(self):
        return self
            
    def __exit__(self, *exc):
        self.close()

then you can write some code like this:

with Resource() as resource:  # __enter__ is called and returns a value as `resource`
    do_something_with_resource()
    # `resource.__exit__` is called

As you have noticed, the resource we get and release is exactly an instance of the class we defined.

What if we hold a resource as an attribute and proxy its __enter__ and __exit__ with __getattr__? We write some code like this:

class ResourceProxy(object):
    def __init__(self):
        self._resource = Resource()

    def __getattr__(self, key):
        return getattr(self._resource, key)

Assuming __getattr__ works fine with __enter__ and __exit__, here is what will happen in with statement:

with ResourceProxy() as resource:  # proxied __enter__ is called
    # now `resource` is NOT a ResourceProxy instance, because what we called is `_resource.__enter__`
    do_something_with_resource()
    # `_resource.__exit__` is called and closed itself properly. 
    # Here is nothing to do with ResourceProxy, because it has never enter `with` context

The behavior above is strange and probably not as the user expected, for the following two reasons:

1. the resource entered into with context is not the object we sent in.
2. when exiting with context, __exit__ method of the proxied object is called, instead of the outer object we sent in. You may think it might help if we add an __exit__ definition on the outer class, but the answer is not, because the outer class has never enter with context.

To conclude, if we make __getattr__ works with __enter__ and __exit__, it will result in bad behaviors. It's not a good design.

Zeba