Mapping over values in a python dictionary

Question

Given a dictionary { k1: v1, k2: v2 ... } I want to get { k1: f(v1), k2: f(v2) ... } provided I pass a function f.

Is there any such built in function? Or do I have to do

dict([(k, f(v)) for (k, v) in my_dictionary.iteritems()])

Ideally I would just write

my_dictionary.map_values(f)

or

my_dictionary.mutate_values_with(f)

That is, it doesn't matter to me if the original dictionary is mutated or a copy is created.

Conakry

A better way of writing your example would be dict((k, f(v)) for k, v in mydict.iteritems()), i.e. without the square brackets, that would prevent the creation of an intermediate list via a generator. – Astronomer

Answer 1

There is no such function; the easiest way to do this is to use a dict comprehension:

my_dictionary = {k: f(v) for k, v in my_dictionary.items()}

Note that there is no such method on lists either; you'd have to use a list comprehension or the map() function.

As such, you could use the map() function for processing your dict as well:

my_dictionary = dict(map(lambda kv: (kv[0], f(kv[1])), my_dictionary.items()))

but that's not that readable, really.

(Note that if you're still using Python 2.7, you should use the .iteritems() method instead of .items() to save memory. Also, the dict comprehension syntax wasn't introduced until Python 2.7.)

Alderman

+1: this is what I would do too. dict(zip(a, map(f, a.values()))) is marginally shorter, but I have to think about what it's doing, and remind myself that yes, keys and values are iterated over in the same order if the dict doesn't change. I don't have to think at all about what the dictcomp is doing, and so it's the right answer. – Technocracy

@DSM: Yeah, the zip(adict, map(f, adict.values()))) trick requires way too much understanding from the casual code reader, not to mention a steady hand in adding all the closing params! :-P – Alderman

{k: f(my_dictionary[k]) for k in my_dictionary} is a little bit shorter, but interestingly it's also a little bit slower (when timing it with timeit, a 500-items dict and str() for f). Don't know why. – Mattress

@chiborg: that's because rather than look up all key-value pairs in one go, you are now using number-of-keys times my_dictionary.__getitem__ calls. – Alderman

Note that since PEP3113 (implemented in python 3.x) tuple parameters are not supported anymore: lambda (k,v): (k, f(v)) is to be rewritten to something like lambda k_v: (k_v[0], f(k_v[1])) – Chyack

@normanius: thanks for the heads-up. Yes, I'm aware that parameter unpacking is gone in Py3 but until searching on Stack Overflow becomes much, much better I don't know in what answers I used the syntax. :-) – Alderman

Why did parameter unpacking get nixed? How is that an improvement ? – Janice

So I read the pep. At least it's admitted: it is not an improvement per se (on the contrary ..) - but a recognition of limitations in tooling for inferring correct types of tuple arguments. – Janice

coming from an FP language, Python would seem incredibly awkward. – Enswathe

Sometimes you've got to apply a function to both the key and the value. E.g., you want to transform the key and the value using the same generated data (faker). Each pair should have their own data generated. In this case, I see no way to use dict comprehensions, only map. – Chrotoem

@Chrotoem so combine them. Map your input iterable to faker objects and then use a dict comp to create the dict key-value pairs. – Alderman

@MartijnPieters Not that easy. My input iterable is a dict. I want to transform it to another dict (both keys and values) using data generated by faker. For each key/value pair a faker object is to be instantiated once. Key is a test input value, value expected result. Both key and value are expressed using some mini language. Like, 's' is to be replaced with a sentence. So no, no room for dict comprehension. – Chrotoem

@Chrotoem sure there is. However, it sounds like you overloaded a dictionary with too much meaning anyway. – Alderman

I've got a function that transforms double newlines into paragraphs, like s\n\ns -> <p>s<p>s, s\r\rs -> <p>s<p>s, and so on. Where s is a sentence. How do I "unload" the dictionary, or how do I use dictionary comprehension when the key and the value must have access to the same random values (sentences used in this test)? – Chrotoem

@x-yuri: this is getting way out of hand for comments. processed = map(nl_to_para, faked_data), result = {key_expr: value_expr for text in processed}. You can put the map() call in the dict comprehension (replace in processed with in map(...)). You can replace the map() with a generator expression. Etc. Iteration is extensible. – Alderman

I hope showing the code will clarify the issue. I don't see a way to avoid using map. But if there is one, I bet it's harder to understand. Then, I'm not sure how people usually go about this, but here's a link to a room to discuss it further (if need be): chat.stackoverflow.com/rooms/195480/… – Chrotoem

@Chyack tuple parameters may not be supported, but lambda parameters are still supported. You just need to unpack the tuples as you map to the lambda function: e.g.) starmap. dict(starmap(lambda k, v: (k, v+1), my_dictionary.items())) – Coffin

Answer 2

These toolz are great for this kind of simple yet repetitive logic.

http://toolz.readthedocs.org/en/latest/api.html#toolz.dicttoolz.valmap

Gets you right where you want to be.

import toolz
def f(x):
  return x+1

toolz.valmap(f, my_list)

Aroid

You know it's a cool library cause of the z. – Freewheel

@Freewheel kewl toolz – Pentalpha

Answer 3

Due to PEP-0469 which renamed iteritems() to items() and PEP-3113 which removed Tuple parameter unpacking, in Python 3.x you should write Martijn Pieters♦ answer like this:

my_dictionary = dict(map(lambda item: (item[0], f(item[1])), my_dictionary.items()))

Misappropriate

Or use the itertools starmap function: my_dictionary = dict(starmap(lambda k, v: (k, f(v)), my_dictionary.items())) – Coffin

Answer 4

You can do this in-place, rather than create a new dict, which may be preferable for large dictionaries (if you do not need a copy).

def mutate_dict(f,d):
    for k, v in d.iteritems():
        d[k] = f(v)

my_dictionary = {'a':1, 'b':2}
mutate_dict(lambda x: x+1, my_dictionary)

results in my_dictionary containing:

{'a': 2, 'b': 3}

Cocklebur

Cool, you should maybe rename mapdict to mutate_values_with or something to make it crystal clear that you rewrite the dict! :) – Conakry

zip(d.keys(), d.values()) works for more versions instead of iteritems() – Tavia

Also, instead of the for loop you could do something similar to what was done in an earlier comment in another answer, dict(zip(d.keys(), [f(v) for v in dict.values()])) – Tavia

Or dict comprehension (not sure if it works on all versions) {k:f(v) for k,v in zip(d.keys(), d.values())} – Tavia

@ytpillai 'zip' or comprehensions make a copy, rather than changing the values in-place, which is the purpose of my answer. The accepted answer is the best one for when a copy is ok. – Cocklebur

My apologies, I didn't realize you wanted to use the items method. However, another improvement to that is possible as well (for non Python 2.7 users) {k:f(v) for k,v in iter(d.items())} – Tavia

Saves space by making an iterator – Tavia

Answer 5

While my original answer missed the point (by trying to solve this problem with the solution to Accessing key in factory of defaultdict), I have reworked it to propose an actual solution to the present question.

Here it is:

class walkableDict(dict):
  def walk(self, callback):
    try:
      for key in self:
        self[key] = callback(self[key])
    except TypeError:
      return False
    return True

Usage:

>>> d = walkableDict({ k1: v1, k2: v2 ... })
>>> d.walk(f)

The idea is to subclass the original dict to give it the desired functionality: "mapping" a function over all the values.

The plus point is that this dictionary can be used to store the original data as if it was a dict, while transforming any data on request with a callback.

Of course, feel free to name the class and the function the way you want (the name chosen in this answer is inspired by PHP's array_walk() function).

Note: Neither the try-except block nor the return statements are mandatory for the functionality, they are there to further mimic the behavior of the PHP's array_walk.

Benildis

This fails to solve the OP question since the __missing__ method won't be called for existing keys, which we want to transform, unless the factory method passed use the origin dict as a fallback somehow, but as that is not part of the example usage, I consider this an unsatisfactory answer to the problem at hand. – Opener

Which existing keys? – Benildis

From the OP: Given a dictionary { k1: v1, k2: v2 ... } .... That is, you already have a dict to begin with.. – Opener

I would like to say that we're both right; but I believe that we're both wrong. You're right in that my answer doesn't answer the question; but not for the reason you invoked. I simply missed the point, giving a way to obtain {v1: f(v1), v2: f(v2), ...} given [v1, v2, ...], and not given a dict. I will edit my answer to correct that. – Benildis

Answer 6

To avoid doing indexing from inside lambda, like:

rval = dict(map(lambda kv : (kv[0], ' '.join(kv[1])), rval.iteritems()))

You can also do:

rval = dict(map(lambda(k,v) : (k, ' '.join(v)), rval.iteritems()))

Sears

That’s a clever manipulation within the 2-tuple itself in the second example. However, it utilizes auto tuple unpacking within the lambda, which is no longer supported in Python 3. Therefore lambda(k,v) will not work. See #21893489 – Badge

Answer 7

Just came accross this use case. I implemented gens's answer, adding a recursive approach for handling values that are also dicts:

def mutate_dict_in_place(f, d):
    for k, v in d.iteritems():
        if isinstance(v, dict):
            mutate_dict_in_place(f, v)
        else:
            d[k] = f(v)

# Exemple handy usage
def utf8_everywhere(d):
    mutate_dict_in_place((
        lambda value:
            value.decode('utf-8')
            if isinstance(value, bytes)
            else value
        ),
        d
    )

my_dict = {'a': b'byte1', 'b': {'c': b'byte2', 'd': b'byte3'}}
utf8_everywhere(my_dict)
print(my_dict)

This can be useful when dealing with json or yaml files that encode strings as bytes in Python 2

Ergotism

Answer 8

• My way to map over dictionary

def f(x): return x+2
bill = {"Alice": 20, "Bob": 10}
d = {map(lambda x: f(x),bill.values())}
print('d: ',dict(d))

Results

: d:  {22: 12}

• Map over iterable in values within dictionary

bills = {"Alice": [20, 15, 30], "Bob": [10, 35]}
d= {map(lambda v: sum(v),bills.values())}
g= dict(map(lambda v: (v[0],sum(v[1])),bills.items()))
# prints
print('d: ',dict(d))
print('g: ',g)

Results

d:  {65: 45}
g:  {'Alice': 65, 'Bob': 45}

Chessman

Answer 9

Suppose we have a dictionary d, we want to map a function over the values of d.

d = {'a': 1,'b': 2, 'c': 3}

def f(x):
    return x + 1

result = dict(zip(d.keys(), map(f, d.values())))
print(result)

The result is

{'a': 2, 'b': 3, 'c': 4}

Berstine