Python dictionary "plus-equal" behavior

Question

I'm trying to understand the exact mechanism behind updating a python dictionary using d[key] += diff. I have some helper classes to trace magic method invocations:

class sdict(dict):
    def __setitem__(self, *args, **kargs):
        print "sdict.__setitem__"
        return super(sdict, self).__setitem__(*args, **kargs)
    def __delitem__(self, *args, **kargs):
        print "sdict.__delitem__"
        return super(sdict, self).__delitem__(*args, **kargs)
    def __getitem__(self, *args, **kargs):
        print "sdict.__getitem__"
        return super(sdict, self).__getitem__(*args, **kargs)
    def __iadd__(self, *args, **kargs):
        print "sdict.__iadd__"
        return super(sdict, self).__iadd__(*args, **kargs)
    def __add__(self, *args, **kargs):
        print "sdict.__add__"
        return super(sdict, self).__add__(*args, **kargs)

class mutable(object):
    def __init__(self, val=0):
        self.value = val
    def __iadd__(self, val):
        print "mutable.__iadd__"
        self.value = self.value + val
        return self
    def __add__(self, val):
        print "mutable.__add__"
        return mutable(self.value + val)

With these tools, let's go diving:

>>> d = sdict()
>>> d["a"] = 0
sdict.__setitem__
>>> d["a"] += 1
sdict.__getitem__
sdict.__setitem__
>>> d["a"]
sdict.__getitem__
1

We don't see any __iadd__ operation invoked here, which makes sense because the left-hand side expression d["a"] returns an integer that does not implement the __iadd__ method. We do see python magically converting the += operator into __getitem__ and __setitem__ calls.

Continuing:

>>> d["m"] = mutable()
sdict.__setitem__
>>> d["m"] += 1
sdict.__getitem__
mutable.__iadd__
sdict.__setitem__
>>> d["m"]
sdict.__getitem__
<__main__.mutable object at 0x106c4b710>

Here the += operator successfully invokes an __iadd__ method. It looks like the += operator is actually being used twice:

• Once for the magic translation to __getitem__ and __setitem__ calls
• A second time for the __iadd__ call.

Where I need help is the following:

• What is the exact, technical mechanism for translating the += operator into __getitem__ and __setitem__ calls?
• In the second example, why is the += operator used twice? Doesn't python translate the statement to d["m"] = d["m"] + 1 (In which case wouldn't we see __add__ be invoked instead of __iadd__?)

Sedimentary

I'm guessing that since the "get the value of" operator has been implemented, the statement is considered equivalent to this: "X = get; X += 1; set(X);" – Goines

This all looks pretty straightforward to me. sdict.__iadd__ is never being invoked because you don't have any code that performs += on an sdict object. You're only performing += on elements in the sdict -- in the first case an integer and in the second case a mutable. But you already demonstrated that you understand this, so I'm not sure what the question is. – Leonardoleoncavallo

Answer 1

In the first example, you didn't apply the += operator to the dictionary. You applied it to the value stored in the d['a'] key, and that's a different object altogether.

In other words, Python will retrieve d['m'] (a __getitem__ call), apply the += operator to that, then set the result of that expression back to d['m'] (the __setitem__ call).

The __iadd__ method either returns self mutated in-place, or a new object, but Python cannot know for sure what the method returned. So it has to call d.__setitem__('m', <return_value_from_d['m'].__iadd__(1)>), always.

The exact same thing happens if you did:

m = d['m']
m += 1
d['m'] = m

but without the extra name m in the global namespace.

If the mutable() instance was not stored in a dictionary but in the global namespace instead, the exact same sequence of events takes place, but directly on the globals() dictionary, and you wouldn't get to see the __getitem__ and __setitem__ calls.

This is documented under the augmented assignment reference documentation:

An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target.

where d['m'] is the target; evaluating the target here involves __getitem__, assigning the result back to the original target invokes __setitem__.

Snoddy

I think my confusion is in the actual mechanism in which python translates += into __getitem__ and __setitem__ calls on the dictionary. In the second example, it seems like it's using the one += operator twice: once to invoke both __getitem__ and __setitem__, and a second time to invoke __iadd__ on mutable. How does that work? – Sedimentary

@moatra: Did you follow the last part of my answer? For += to work, Python needs to retrieve an object on the left-hand side of the equation first (__getitem__), then apply +=, then when that is done, put the object back where it came from (__setitem__). – Snoddy

My current understanding is that Python translates d["m"] += 1 to d["m"] = d["m"] + 1. Does it instead translate it to d["m"] = d["m"] += 1? Regardless, where is this behavior specified? – Sedimentary

Your understanding is incorrect. See the augmented assignment documentation. – Snoddy

If no __iadd__ method is defined on d['m'], then d['m'] = d['m'] + 1 would be applied, but that's not the case here. – Snoddy

Ah. My confusion stemmed from incorrectly thinking += was implemented purely by a magic method --- it's actually a grammar construct. – Sedimentary

@MartijnPieters I swear I've never seen anyone being such a pro @ python ever, period. I would thumbs up x1000 at each answer of yours if I could. Thank you sir! I hope you never retire! – Rationalism

Answer 2

Because, as specified in docs, __iadd__ might perform the operation in-place but the result will be either self or a new object, thus __setitem__ is called.

Rumble