How to implement a lazy setdefault?

Question

One minor annoyance with dict.setdefault is that it always evaluates its second argument (when given, of course), even when the first argument is already a key in the dictionary.

For example:

import random
def noisy_default():
    ret = random.randint(0, 10000000)
    print 'noisy_default: returning %d' % ret
    return ret

d = dict()
print d.setdefault(1, noisy_default())
print d.setdefault(1, noisy_default())

This produces ouptut like the following:

noisy_default: returning 4063267
4063267
noisy_default: returning 628989
4063267

As the last line confirms, the second execution of noisy_default is unnecessary, since by this point the key 1 is already present in d (with value 4063267).

Is it possible to implement a subclass of dict whose setdefault method evaluates its second argument lazily?

---

EDIT:

Below is an implementation inspired by BrenBarn's comment and Pavel Anossov's answer. While at it, I went ahead and implemented a lazy version of get as well, since the underlying idea is essentially the same.

class LazyDict(dict):
    def get(self, key, thunk=None):
        return (self[key] if key in self else
                thunk() if callable(thunk) else
                thunk)


    def setdefault(self, key, thunk=None):
        return (self[key] if key in self else
                dict.setdefault(self, key,
                                thunk() if callable(thunk) else
                                thunk))

Now, the snippet

d = LazyDict()
print d.setdefault(1, noisy_default)
print d.setdefault(1, noisy_default)

produces output like this:

noisy_default: returning 5025427
5025427
5025427

Notice that the second argument to d.setdefault above is now a callable, not a function call.

When the second argument to LazyDict.get or LazyDict.setdefault is not a callable, they behave the same way as the corresponding dict methods.

If one wants to pass a callable as the default value itself (i.e., not meant to be called), or if the callable to be called requires arguments, prepend lambda: to the appropriate argument. E.g.:

d1.setdefault('div', lambda: div_callback)

d2.setdefault('foo', lambda: bar('frobozz'))

Those who don't like the idea of overriding get and setdefault, and/or the resulting need to test for callability, etc., can use this version instead:

class LazyButHonestDict(dict):
    def lazyget(self, key, thunk=lambda: None):
        return self[key] if key in self else thunk()


    def lazysetdefault(self, key, thunk=lambda: None):
        return (self[key] if key in self else
                self.setdefault(key, thunk()))

Canoodle

You can't make it not evaluate the second argument. What you'd have to do is wrap that argument in a function (e.g., with lambda) and then have setdefault call the function only if needed. – Anthologize

Can I suggest you add *args, **kwargs to the signatures of lazyget, lazysetdefault, and the call to thunk()? This would allow your lazy stuff to take parameters. e.g. lbd.lazysetdefault('total', sum, [1, 2, 3, 4], start=2) – Brettbretz

Answer 1

No, evaluation of arguments happens before the call. You can implement a setdefault-like function that takes a callable as its second argument and calls it only if it is needed.

Sochi

Answer 2

This can be accomplished with defaultdict, too. It is instantiated with a callable which is then called when a nonexisting element is accessed.

from collections import defaultdict

d = defaultdict(noisy_default)
d[1] # noise
d[1] # no noise

The caveat with defaultdict is that the callable gets no arguments, so you can not derive the default value from the key as you could with dict.setdefault. This can be mitigated by overriding __missing__ in a subclass:

from collections import defaultdict

class defaultdict2(defaultdict):
    def __missing__(self, key):
        value = self.default_factory(key)
        self[key] = value
        return value

def noisy_default_with_key(key):
    print key
    return key + 1

d = defaultdict2(noisy_default_with_key)
d[1] # prints 1, sets 2, returns 2
d[1] # does not print anything, does not set anything, returns 2

For more information, see the collections module.

Supat

Answer 3

You can do that in a one-liner using a ternary operator:

value = cache[key] if key in cache else cache.setdefault(key, func(key))

If you are sure that the cache will never store falsy values, you can simplify it a little bit:

value = cache.get(key) or cache.setdefault(key, func(key))

Spunky

If you're checking key in dict there's no point in using setdeault – Listed

This will require to search key in cache twice. Which is not a big deal for dict based on Hash-Map, but still doesn't make so mach sense. – Coleville

@Listed If you don't call setdefault the cache won't be updated. setdefault is both setting the empty cache and returning its value at the same time – Spunky

Not looking up the key twice would be a micro-optimisation from O( "2" n) to O(n). Python's probably the wrong choice if this is an issue. #25778214 – Duodenum

@cz "Not looking up the key twice would be a micro-optimisation from O( "2" n) to O(n)", dicts are supposed to be O(1) for most purposes (assuming no collisions). – Bunk

Answer 4

There seems to be no one-liner that doesn't require an extra class or extra lookups. For the record, here is a easy (even not concise) way of achieving that without either of them.

try:
    value = dct[key]
except KeyError:
    value = noisy_default()
    dct[key] = value
return value

Donation

Avoids double lookup, and is completely understandable. Could even be a line shorter using; value = dct[key] = noisy_default() – Freitas

Answer 5

For Python 3.8+ the best I can come with is a function.

from typing import MutableMapping, Callable, TypeVar

K = TypeVar('K')
V = TypeVar('V')
MISSING = object()


def setdefault_lazy(d: MutableMapping[K ,V], key: K, func: Callable[[], V]) -> V:
    if (value := d.get(key, MISSING)) is MISSING:
        d[key] = value = func()
    return value

Use case:

d = dict()
print(setdefault_lazy(d, 1, noisy_default))
print(setdefault_lazy(d, 1, noisy_default))

You can make your own dict class which uses this function like so:

class MyDict(dict):
    setdefault_lazy = setdefault_lazy

d = MyDict(name="John")
print(d.setdefault_lazy('age', noisy_default))
print(d.setdefault_lazy('age', noisy_default))

Bohlen