Using both __setattr__ and descriptors for a python class

Question

I'm writing a python class that uses __setattr__ and __getattr__ to provide custom attribute access.

However, some attributes can't be handled in a generic way, so I was hoping to use descriptors for those.

A problem arises in that for a descriptor, the descriptor's __get__ will be invoked in favour of the instances __getattr__, but when assigning to an attribute, __setattr__ will be invoked in favour of the descriptors __set__.

An example:

class MyDesc(object):

    def __init__(self):
        self.val = None

    def __get__(self, instance, owner):
        print "MyDesc.__get__"
        return self.val

    def __set__(self, instance, value):
        print "MyDesc.__set__"
        self.val = value

class MyObj(object):

    foo = MyDesc()

    def __init__(self, bar):
        object.__setattr__(self, 'names', dict(
            bar=bar,
        ))
        object.__setattr__(self, 'new_names', dict())

    def __setattr__(self, name, value):
        print "MyObj.__setattr__ for %s" % name
        self.new_names[name] = value

    def __getattr__(self, name):
        print "MyObj.__getattr__ for %s" % name

        if name in self.new_names:
            return self.new_names[name]

        if name in self.names:
            return self.names[name]

        raise AttributeError(name)

if __name__ == "__main__":
    o = MyObj('bar-init')

    o.bar = 'baz'
    print o.bar

    o.foo = 'quux'
    print o.foo

prints:

MyObj.__setattr__ for bar
MyObj.__getattr__ for bar
baz
MyObj.__setattr__ for foo
MyDesc.__get__
None

The descriptor's __set__ is never called.

Since the __setattr__ definition isn't just overriding behaviour for a limited set of names, there's no clear place that it can defer to object.__setattr__

Is there a recommended way to have assigning to attributes use the descriptor, if available, and __setattr__ otherwise?

Approbate

Answer 1

update

Revisiting this answer more than 10 years later, I realized that I included a recipe, but did not explain the outstanding behavior the OP met: a descritpor's __get__ is called before an instance's __getattr__, yes. But the actual converse to __setattr__ is __getattribute__, not __getattr__. Inside object's __getattribute__ is where is the code with the search order for attributes in first place, and the code responsible for calling a descriptor's __get__ itself. __getattr__ is called by the code in __getattribute__ itself as a last resort, after inspecting any possible descriptors, __slots__, and the instance's __dict__.

original answer: just do what the OP wants to achieve

I think I'd approach this by having a mechanism to automatically mark which are the descriptors in each class, and wrap the __setattr__ in a way that it'd call object's normal behavior for those names.

This can be easily achieved with a metaclass (and a decorator for __setattr__

def setattr_deco(setattr_func):
    def setattr_wrapper(self, attr, value):
        if attr in self._descriptors:
            return object.__setattr__(self, attr, value)
        return setattr_func(self, attr, value)
    return setattr_wrapper

class MiscSetattr(type):
    def __new__(metacls, name, bases, dct):
        descriptors = set()
        for key, obj in dct.items():
            if key == "__setattr__":
                dct[key] = setattr_deco(obj)
            elif hasattr(obj, "__get__"):
                descriptors.add(key)
        dct["_descriptors"] = descriptors
        return type.__new__(metacls, name, bases, dct)
    
# and use MiscSetattr as metaclass for your classes

Potheen

I may be not following this, but why is __setattr__ called for the descriptors? Do we not want __set__ to be called? – Factfinding

__setattr__ is how one can customize attribute writing access to any class - all attribute setting go through it, even when the target attribute is a descriptor. Actually, calling the checking for a data descriptor and calling its __set__ is performed by the default __setattr__ code in the object class: Python is very consistent in this way, providing several points where one can customize things like attribute access, but in a way once you choose which way you will customize, you can do that in a single place. (No having to customize both __setattr__ and a __set__. – Potheen

Answer 2

One of possible ways:

def __setattr__(self, name, value):
    print "MyObj.__setattr__ for %s" % name
    for cls in self.__class__.__mro__ + (self, ):
        if name in cls.__dict__:
            return object.__setattr__(self, name, value)
    print 'New name', name, value
    self.new_names[name] = value

It checks if name already defined in class, base classes or instance and then it calls object.__setattr__ which will execute descriptor __set__.

Another way:

def __setattr__(self, name, value):
    print "MyObj.__setattr__ for %s" % name
    try:
        object.__getattribute__(self, name)
    except AttributeError:
        print 'New name', name, value
        self.new_names[name] = value
    else:
        object.__setattr__(self, name, value)

But it will call descriptor's __get__.

P.S.

I'm not sure about need to check all __mro__ members since MyObj will contain inherited class members in __dict__.

Maybe for cls in (self.__class__, self):... will be enough.

Matriculate

You're second way won't work. In my example, you're allowed to set new attributes on the object that didn't already exist, and this method would disallow that. – Approbate

@SpoonMeiser, I think it works like your's. Take a look ideone.com/HMBcc. Maybe I'm misssing something. – Matriculate

If I did o.quux = 3, then o.__dict__['quux'] would be set instead of o.new_names['quux'] – Approbate

@SpoonMeiser, I can't reproduce. What is your Python version? ideone.com/XXAR5 – Matriculate

There is a problem though, if __get__ might sometimes raise AttributeError – Approbate