How do I implement __getattribute__ without an infinite recursion error?
Asked Answered
C

6

144

I want to override access to one variable in a class, but return all others normally. How do I accomplish this with __getattribute__?

I tried the following (which should also illustrate what I'm trying to do) but I get a recursion error:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return self.__dict__[name]

>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp
Collective answered 16/12, 2008 at 16:4 Comment(0)
C
161

You get a recursion error because your attempt to access the self.__dict__ attribute inside __getattribute__ invokes your __getattribute__ again. If you use object's __getattribute__ instead, it works:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return object.__getattribute__(self, name)

This works because object (in this example) is the base class. By calling the base version of __getattribute__ you avoid the recursive hell you were in before.

Ipython output with code in foo.py:

In [1]: from foo import *

In [2]: d = D()

In [3]: d.test
Out[3]: 0.0

In [4]: d.test2
Out[4]: 21

Update:

There's something in the section titled More attribute access for new-style classes in the current documentation, where they recommend doing exactly this to avoid the infinite recursion.

Cherise answered 16/12, 2008 at 16:26 Comment(16)
Interesting. So what are you doing there? Why would object have my variables?Collective
..and do I want to always use object? What if I inherit from other classes?Collective
Because it's the parent class. You're talking to the base class's version of __getattribute__() rather than your new shiny (and recursion-addled) __getattribute__()Spoke
Use the class you're inheriting from. Whatever that might be.Spoke
If you look at the first argument to object's function, it is self. Self is the current instance of your class, the object. You use it in a similar manner when you subclass other classes and you need to call their constructur, initCherise
Ah so I'm only taking the function getattribute from the base class, the values are still coming from my class. So really you can use any class up the hierarchy?Collective
Though I guess object will have a known good getattribute?Collective
Yes, each time you create a class and you don't write your own, you use the getattribute supplied by object.Cherise
Egil: That's not an entirely safe way to look at it. If I'm using a class that inherits another class, that inherits another class that inherits object (and I didn't write any of the inherited classes) If one of those overrides getattribute and you call object's version, it might not work as ...Spoke
... expected. You should always call the next rung up the inheritance ladder (unless you know what you're doing).Spoke
But I see that my comment can be misleading.Cherise
@Oli, @Egil, I don't understand why `pass 2 args to object's getattr like this, object.__getattribute__(self, name)? getattr is not staticmethod right?Enterpriser
@Alcott: No, __getattribute__() is not a staticmethod, which is precisely why, when calling the unbound object class's version, it is necessary to pass the instance argument self in addition to the attribute name.Jovian
isn't it better to use super() and thus use the first getattribute method python finds in your base classes? - super(D, self).__getattribute__(name)Sulphuric
Or you can just use super().__getattribute__(name) in Python 3.Clerkly
Using object.__getattribute__(self, name) is a very bad idea if the super-class of self is anything other than objectLicentious
E
35

Actually, I believe you want to use the __getattr__ special method instead.

Quote from the Python docs:

__getattr__( self, name)

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.

Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.

Embowed answered 16/12, 2008 at 22:1 Comment(1)
Actually, according to the nature of the OP's code, overriding __getattr__ for test would be useless, since it would always find it "in the usual places".Bedelia
P
21

Python language reference:

In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, object.__getattribute__(self, name).

Meaning:

def __getattribute__(self,name):
    ...
        return self.__dict__[name]

You're calling for an attribute called __dict__. Because it's an attribute, __getattribute__ gets called in search for __dict__ which calls __getattribute__ which calls ... yada yada yada

return  object.__getattribute__(self, name)

Using the base classes __getattribute__ helps finding the real attribute.

Pru answered 16/12, 2008 at 16:39 Comment(0)
A
14

How is the __getattribute__ method used?

It is called before the normal dotted lookup. If it raises AttributeError, then we call __getattr__.

Use of this method is rather rare. There are only two definitions in the standard library:

$ grep -Erl  "def __getattribute__\(self" cpython/Lib | grep -v "/test/"
cpython/Lib/_threading_local.py
cpython/Lib/importlib/util.py

Best Practice

The proper way to programmatically control access to a single attribute is with property. Class D should be written as follows (with the setter and deleter optionally to replicate apparent intended behavior):

class D(object):
    def __init__(self):
        self.test2=21

    @property
    def test(self):
        return 0.

    @test.setter
    def test(self, value):
        '''dummy function to avoid AttributeError on setting property'''

    @test.deleter
    def test(self):
        '''dummy function to avoid AttributeError on deleting property'''

And usage:

>>> o = D()
>>> o.test
0.0
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0

A property is a data descriptor, thus it is the first thing looked for in the normal dotted lookup algorithm.

Options for __getattribute__

You several options if you absolutely need to implement lookup for every attribute via __getattribute__.

  • raise AttributeError, causing __getattr__ to be called (if implemented)
  • return something from it by
    • using super to call the parent (probably object's) implementation
    • calling __getattr__
    • implementing your own dotted lookup algorithm somehow

For example:

class NoisyAttributes(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self, name):
        print('getting: ' + name)
        try:
            return super(NoisyAttributes, self).__getattribute__(name)
        except AttributeError:
            print('oh no, AttributeError caught and reraising')
            raise
    def __getattr__(self, name):
        """Called if __getattribute__ raises AttributeError"""
        return 'close but no ' + name    


>>> n = NoisyAttributes()
>>> nfoo = n.foo
getting: foo
oh no, AttributeError caught and reraising
>>> nfoo
'close but no foo'
>>> n.test
getting: test
20

What you originally wanted.

And this example shows how you might do what you originally wanted:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return super(D, self).__getattribute__(name)

And will behave like this:

>>> o = D()
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
>>> del o.test

Traceback (most recent call last):
  File "<pyshell#216>", line 1, in <module>
    del o.test
AttributeError: test

Code review

Your code with comments. You have a dotted lookup on self in __getattribute__. This is why you get a recursion error. You could check if name is "__dict__" and use super to workaround, but that doesn't cover __slots__. I'll leave that as an exercise to the reader.

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:      #   v--- Dotted lookup on self in __getattribute__
            return self.__dict__[name]

>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp
Ambrose answered 5/2, 2015 at 23:48 Comment(0)
A
13

Are you sure you want to use __getattribute__? What are you actually trying to achieve?

The easiest way to do what you ask is:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    test = 0

or:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    @property
    def test(self):
        return 0

Edit: Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.

Edit2: Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    _test = 0

    def get_test(self):
        return self._test

    def set_test(self, value):
        self._test = value

    test = property(get_test, set_test)

Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.

Anitaanitra answered 16/12, 2008 at 16:32 Comment(2)
Oh that doesn't quiet work when you run the class though, no? File "Script1.py", line 5, in init self.test = 20 AttributeError: can't set attributeCollective
True. I'll fix that up as a third example. Well spotted.Anitaanitra
T
5

Here is a more reliable version:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21
    def __getattribute__(self, name):
        if name == 'test':
            return 0.
        else:
            return super(D, self).__getattribute__(name)

It calls __getattribute__ method from parent class, eventually falling back to object.__getattribute__ method if other ancestors don't override it.

Tactless answered 29/5, 2013 at 10:18 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.