Python: inconsistence in the way you define the function __setattr__?

Question

Consider this code:

class Foo1(dict):
    def __getattr__(self, key): return self[key]
    def __setattr__(self, key, value): self[key] = value

class Foo2(dict):
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__

o1 = Foo1()
o1.x = 42
print(o1, o1.x)

o2 = Foo2()
o2.x = 42
print(o2, o2.x)

I would expect the same output. However, with CPython 2.5, 2.6 (similarly in 3.2) I get:

({'x': 42}, 42)
({}, 42)

With PyPy 1.5.0, I get the expected output:

({'x': 42}, 42)
({'x': 42}, 42)

Which is the "right" output? (Or what should be the output according to the Python documentation?)

---

Here is the bug report for CPython.

Prude

Interesting puzzle. I wonder if CPython's behaviour has something to do with dict.__setitem__ being a slot wrapper. – Fixing

Answer 1

I suspect it has to do with a lookup optimization. From the source code:

 /* speed hack: we could use lookup_maybe, but that would resolve the
       method fully for each attribute lookup for classes with
       __getattr__, even when the attribute is present. So we use
       _PyType_Lookup and create the method only when needed, with
       call_attribute. */
    getattr = _PyType_Lookup(tp, getattr_str);
    if (getattr == NULL) {
        /* No __getattr__ hook: use a simpler dispatcher */
        tp->tp_getattro = slot_tp_getattro;
        return slot_tp_getattro(self, name);
    }

The fast path does does not look it up on the class dictionary.

Therefore, the best way to get the desired functionality is to place an override method in the class.

class AttrDict(dict):
    """A dictionary with attribute-style access. It maps attribute access to
    the real dictionary.  """
    def __init__(self, *args, **kwargs):
        dict.__init__(self, *args, **kwargs)

    def __repr__(self):
        return "%s(%s)" % (self.__class__.__name__, dict.__repr__(self))

    def __setitem__(self, key, value):
        return super(AttrDict, self).__setitem__(key, value)

    def __getitem__(self, name):
        return super(AttrDict, self).__getitem__(name)

    def __delitem__(self, name):
        return super(AttrDict, self).__delitem__(name)

    __getattr__ = __getitem__
    __setattr__ = __setitem__

     def copy(self):
        return AttrDict(self)

Which I found works as expected.

Unknowable

Answer 2

It's a known (and maybe not so well) documented difference. PyPy does not differentiate between functions and builtin functions. In CPython functions get binded as unbound methods when stored on the class (the have __get__), while builtin functions don't (they're different).

Under PyPy however, builtin functions are exactly the same as python functions, so the interpreter can't tell them apart and treats them as python-level functions. I think this was defined as implementation details, although there was some discussion on python-dev about removing this particular difference.

Cheers,
fijal

Theall

Answer 3

Note the following:

>>> dict.__getitem__ # it's a 'method'
<method '__getitem__' of 'dict' objects> 
>>> dict.__setitem__ # it's a 'slot wrapper'
<slot wrapper '__setitem__' of 'dict' objects> 

>>> id(dict.__dict__['__getitem__']) == id(dict.__getitem__) # no bounding here
True
>>> id(dict.__dict__['__setitem__']) == id(dict.__setitem__) # or here either
True

>>> d = {}
>>> dict.__setitem__(d, 1, 2) # can be called directly (since not bound)
>>> dict.__getitem__(d, 1)    # same with this
2

Now we can just wrap them (and __getattr__ will work even without that):

class Foo1(dict):
    def __getattr__(self, key): return self[key]
    def __setattr__(self, key, value): self[key] = value

class Foo2(dict):
    """
    It seems, 'slot wrappers' are not bound when present in the __dict__ 
    of a class and retrieved from it via instance (or class either).
    But 'methods' are, hence simple assignment works with __setitem__ 
    in your original example.
    """
    __setattr__ = lambda *args: dict.__setitem__(*args)
    __getattr__ = lambda *args: dict.__getitem__(*args) # for uniformity, or 
    #__getattr__ = dict.__getitem__                     # this way, i.e. directly


o1 = Foo1()
o1.x = 42
print(o1, o1.x)

o2 = Foo2()
o2.x = 42
print(o2, o2.x)

Which gives:

>>>
({'x': 42}, 42)
({'x': 42}, 42)

The mechanism behind the behavior in question is (probably, I'm no expert) outside the 'clean' subset of Python (as documented in thorough books like 'Learning Python' or 'Python in a nutshell' and somewhat loosely specified at python.org) and pertains to the part of the language that is documented 'as-is' by the implementation (and is subject to (rather) frequent changes).

Asymmetry

Answer 4

Simple, addresses the subdicts and correct AttributeError, although being very small:

class DotDict(dict):
    def __init__(self, d: dict = {}):
        super().__init__()
        for key, value in d.items():
            self[key] = DotDict(value) if type(value) is dict else value
    
    def __getattr__(self, key):
        if key in self:
            return self[key]
        raise AttributeError(key) #Set proper exception, not KeyError

    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

Pumpkinseed