Why does Python 3 need dict.items to be wrapped with list()?

Question

I'm using Python 3. I've just installed a Python IDE and I am curious about the following code warning:

features = { ... }
for k, v in features.items():
    print("%s=%s" % (k, v))

Warning is: "For Python3 support should look like ... list(features.items()) "

Also there is mention about this at http://docs.python.org/2/library/2to3.html#fixers

It also wraps existing usages of dict.items(), dict.keys(), and dict.values() in a call to list.

Why is this necessary?

Jeffreyjeffreys

I think that your IDE should instead warn you that '{}={}'.format(k, v) is recommended instead. :) Reference: docs.python.org/3.0/whatsnew/… – Manganate

Answer 1

You can safely ignore this "extra precautions" warning: your code will work the same even without list in both versions of Python. It would run differently if you needed a list (but this is not the case): in fact, features.items() is a list in Python 2, but a view in Python 3. They work the same when used as an iterable, as in your example.

Now, the Python 2 to Python 3 conversion tool 2to3 errs on the side of safety, and assumes that you really wanted a list when you use dict.items(). This may not be the case (as in the question), in which case dict.items() in Python 3 (no wrapping list) is better (faster, and less memory-consuming, since no list is built).

Concretely, this means that Python 2 code can explicitly iterate over the view: for k, v in features.viewitems() (which will be converted in Python 3 by 2to3 to features.items()). It looks like your IDE thinks that the code is Python 2, because your for statement is very good, in Python 3, so there should be no warning about Python 3 support.

Manganate

Note: in general, it is not safe to ignore the warning if the loop modifies the dict. It breaks on Python 3 without list(). – Skyeskyhigh

This should never happen: a loop normally cannot modify the dictionary it is iterating over; this generally raises a RuntimeError: dictionary changed size during iteration (both in Python 2 and Python 3). Reference: docs.python.org/dev/whatsnew/2.7.html#pep-3106-dictionary-views – Manganate

d.items() returns a list on Python 2 therefore the loop can modify the dictionary. Just try it: d = {1:2}; for k, v in d.items(): d[3]=4 - this code works on Python 2 but it breaks on Python 3 with the error that you've mentioned. – Skyeskyhigh

Indeed. I had the likes of my_dict.iteritems() in mind (which does raise the exception I mentioned, even in Python 2): in the case you cite, one is not "iterating over the dictionary" but over a list, as you say. That said, now I see what you meant: that list() is necessary in Python 3 if one wants to modify the dictionary. I had missed this because (1) the warning is actually a little strange, as the for loop statement is perfect Python 3 code; and (2) this is a relatively rare situation (never modified a dictionary after items() in 7 years!). Good point. :) – Manganate

This doesn't explain why a list is necessary, which is required by the main question. – Ophthalmitis

Can you elaborate on this? The answer to the original question is that list() is not necessary. A longer answer is that if the features dictionary of the original question were modified in the loop, then list() would effectively freeze the list of items and the dictionary could be modified in the loop (iterating over a view of the items in a dictionary that changes may raise an exception). – Manganate

The key phrase in this answer is when used as an iterable – Youlandayoulton

I know this is an old post but I just wanted to point out that having to wrap every items() call in a list feels extremely not pythonic. – Latashalatashia

What do you mean exactly? There is no need to do this in Python 2, and in Python 3, most of the time there isn't either. But when in Python you do need to obtain a list from the items, then you have to wrap them in list() but this is by definition "Pythonic": it just means that you know that items() returns an iterable, and that if you want to build a list from it, Python lets you do it. – Manganate

Answer 2

In Python 2, the methods items(), keys() and values() used to "take a snapshot" of the dictionary contents and return it as a list. It meant that if the dictionary changed while you were iterating over the list, the contents in the list would not change.

In Python 3, these methods return a view object whose contents change dynamically as the dictionary changes. Therefore, in order for the behavior of iterations over the result of these methods to remain consistent with previous versions, an additional call to list() has to be performed in Python 3 to "take a snapshot" of the view object contents.

Delusive

Thank you for the answer and (+1) of course, but this derive one more question on it: ... change dynamically ... If I'm ok with dynamic changes, do Python provides safe iteration over it (just compare with java behavior - raise exception from iterator on changing container) – Jeffreyjeffreys

@Dewfy, not really. The view objects themselves are safe, but as soon as you create iterators from them and use these iterators in loops, adding or removing entries from the source dictionary may lead to errors or "miss" entries. The documentation says iterating views while adding or deleting entries in the dictionary may raise a RuntimeError or fail to iterate over all entries. – Bluey

This should be the accepted answer, since it actually answers the question. – Ophthalmitis

Answer 3

Python 3 returns a Dictionary View Object rather than a list which Python 2 would return and some operators that you would expect may not be true - also a View Object will change if the underlying dictionary changes, (possibly in the code that you are iterating through which could cause some unwelcome surprises).

Polinski

Answer 4

In Python 3, dict.items(), dict.keys(), and dict.values() are iterators. Therefore if you are expecting a list, you might get some errors when doing operations that work on lists, but not necessarily on iterators, such as len(dict.items()) (will generate a TypeError).

CORRECTION

The dict_items returned by calling dict.items() in Python 3 does indeed have a __len__() and will not generate a TypeError. The dict_items object is not a list, however, and does not have list methods, such as append(), index(), etc...

Also, as the other (I would say much better) answers by Hamidi and Barnes state, dict_items is a view object that will dynamically change when the dict is altered.

Lezlielg

Nitpick: It is not technically correct that dict.items() etc. are iterators in Python 3: they are iterable, meaning that they can be converted into iterators. Reference: docs.python.org/2/glossary.html – Manganate

Major Nitpick - len(dict.items()) should be len(dictview) which is specifically given as an example in the documentation, docs.python.org/3/library/stdtypes.html#dict.items, while this answer incorrectly states that it will NOT work and will generate a type error. – Polinski

@SteveBarnes: You're absolutely right. I should have checked the docs before attempting to answer. – Lezlielg

I'm surprised that my answer, which was just plain wrong, actually received upvotes. – Lezlielg

Very Gracious @JoelCornett – Polinski

Answer 5

When converting a project to python 3 using 2to3, you can disable this by excluding the dict fixer for more concise output:

$ 2to3 -x dict *

Watch out for iteritems(), iterkeys() https://docs.python.org/2/library/2to3.html#2to3fixer-dict and fix by hand.

Thematic

Question is not about "how to close an eye", it was about the root cause – Jeffreyjeffreys

True, I just shared this because in retrospect I wish I knew it earlier. – Thematic