Descriptors as instance attributes in python

Question

To the question:

Why can't descriptors be instance attributes?

it has been answered that:

descriptor objects needs to live in the class, not in the instance

because that is the way that the __getattribute__ is implemented.

A simple example. Consider a descriptor:

class Prop(object):

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj._value * obj._multiplier

    def __set__(self, obj, value):
        if obj is None:
            return self
        obj._value = value

class Obj(object):

    val = Prop()

    def __init__(self):
        self._value = 1
        self._multiplier = 0

Consider the case in which each obj has multiple Prop: I would need to use unique names to identify the values and multipliers (Like here. Having a per instance descriptor object would allow to store the _multiplier (and the _value) in the descriptor itself, simplifying a few things.

To implement per instance descriptor attributes you need to either:

1. create a per instance class See here
2. override __getattribute__ See here

I am aware that similar questions have been raised before, but I have not found a real explanation:

1. Why Python is designed this way?
2. What is the suggested way to store information that the descriptor needs but is per instance?

Awed

Just FYI, here it is clearly mentioned in the documentation – Liquidity

Answer 1

Plenty of advanced functionality only works when defined on a class rather than an instance; all of the special methods, for example. As well as making code evaluation more efficient, this makes clear the separation between instances and types which otherwise would tend to collapse (because of course all types are objects).

I'm not sure how recommended this is, but you could in the instance store a mapping from descriptor instance to attribute value:

class Prop(object):
     def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj._value * obj._multiplier[self]

    def __set__(self, obj, value):
        if obj is None:
            return self
        obj._value = value

class Obj(object):
    val = Prop()

    def __init__(self):
        self._value = 1
        self._multiplier = {Obj.val: 0}

This has obvious advantages over the other two suggested options:

1. per-instance classes break object orientation and increase memory usage;
2. overriding __getattribute__ is inefficient (as all attribute access must go through the overridden special method) and is fragile.

---

As an alternative, you could use a proxy property:

class PerInstancePropertyProxy(object):
    def __init__(self, prop):
        self.prop = prop
    def __get__(self, instance, owner):
        if instance is None:
            return self
        return instance.__dict__[self.prop].__get__(instance, owner)
    def __set__(self, instance, value):
        instance.__dict__[self.prop].__set__(instance, value)
class Prop(object):
    def __init__(self, value, multiplier):
        self.value = value
        self.multiplier = multiplier
    def __get__(self, instance, owner):
        if instance is None:
            return self
        return self.value * self.multiplier
    def __set__(self, instance, value):
        self.value = value
class Obj(object):
    val = PerInstancePropertyProxy('val')
    def __init__(self):
        self.__dict__['val'] = Prop(1.0, 10.0)
    def prop(self, attr_name):
        return self.__dict__[attr_name]

Tiptoe

Can you expand what you mean by break object orientation. Second, the problem that I see with this approach is how to provide a simple API to change the multiplier. Users will have to do something like obj._multiplier[Obj.val] = 10. This can be wrapped in a function def change_multiplier(self, attr_name, new_value) but does not scale nicely if Prop has multiple attributes. Something like def prop(self, attr_name): return self.__dict__[attr_name] could be used to do something like obj.prop('val').multiplier = 10. – Awed

@Awed there's a usual assumption that instances have the same type; violate that and various things will break. As far as changing the multiplier goes, perhaps a proxy property? - see edit above. – Tiptoe

Is true that all instances will not be of the same type, but you might make subclasses so isinstance will still work. Regarding the proxy, I wrote something like this but I am not sure that is a good idea. Basically, obj.prop('val') returns a proxy object that knows about obj and val. When you do obj.prop('val').multiplier = 10 it writes to obj._multiplier[val] = 10. I am just not sure how maintainable it will be. – Awed

Answer 2

This exact question was raised on Python-list earlier this year. I'm just going to quote Ian G. Kelly's response:

The behavior is by design. First, keeping object behavior in the class definition simplifies the implementation and also makes instance checks more meaningful. To borrow your Register example, if the "M" descriptor is defined by some instances rather than by the class, then knowing that the object "reg" is an instance of Register does not tell me anything about whether "reg.M" is a valid attribute or an error. As a result, I'll need to guard virtually every access of "reg.M" with a try-except construct just in case "reg" is the wrong kind of register.

Second, the separation of class from instance also helps you keep object behavior separate from object data. Consider the following class:

class ObjectHolder(object):
    def __init__(self, obj):
        self.obj = obj

Don't worry about what this class might be useful for. Just know that it's meant to hold and provide unrestricted access to arbitrary Python objects:

>>> holder = ObjectHolder(42)
>>> print(holder.obj) 42
>>> holder.obj = range(5)
>>> print(holder.obj) [0, 1, 2, 3, 4]

Since the class is meant to hold arbitrary objects, it's even valid that somebody might want to store a descriptor object there:

>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj) <property object at 0x02415AE0>

Now suppose that Python invoked the descriptor protocol for descriptors stored in instance attributes:

>>> holder = ObjectHolder(None)
>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'ObjectHolder' object has no attribute 'foo'

In this case, the ObjectHolder would fail to simply hold the property object as data. The mere act of assigning the property object, a descriptor, to an instance attribute would change the behavior of the ObjectHolder. Instead of treating "holder.obj" as a simple data attribute, it would start invoking the descriptor protocol on accesses to "holder.obj" and ultimately redirect them to the non-existent and meaningless "holder.foo" attribute, which is certainly not what the author of the class intended.

If you want to be able to support multiple instances of a descriptor, just make that descriptor's constructor take a name argument (prefix), and prefix the added attributes with that name. You could even create a namespace object (dictionary) within the class instance to hold all of the new property instances.

Levigate

Answer 3

In Python 3.6 this can be done quite easily. Maybe it's not as intended but hey, if it works, right? ;)

Python 3.6 adds the __set_name__ method:

object.__set_name__(self, owner, name)

Called at the time the owning class owner is created. The descriptor has been assigned to name.

New in version 3.6.

Using this name to store the internal value in the instance's dict seems to work fine.

>>> class Prop:
...     def __set_name__(self, owner, name):
...         self.name = name
...     def __get__(self, instance, owner):
...         print('get')
...         return instance.__dict__.setdefault(self.name, None)
...     def __set__(self, instance, value):
...         print('set')
...         instance.__dict__[self.name] = value
... 
>>> class A:
...     prop = Prop()
... 
>>> a = A()
>>> a.prop = 'spam'
set
>>> a.prop
get
'spam'

Note that this is not a full descriptor implementation and of course if you decide to use it it's at your own risk.

Beetner

Question: What do you mean by this is not a full descriptor implementation? To my knowledge, it is a perfectly applicable example of a descriptor. – Hasty

instance.__dict__ – what about classes with __slots__? – Bathyal