Sorting list by an attribute that can be None

Question

I'm trying to sort a list of objects using

my_list.sort(key=operator.attrgetter(attr_name))

but if any of the list items has attr = None instead of attr = 'whatever',

then I get a TypeError: unorderable types: NoneType() < str()

In Py2 it wasn't a problem. How do I handle this in Py3?

Pasqualepasqueflower

have you tried 2to3? – Hbeam

Yes. That is a piece of code that stopped working after 2to3'ing it. – Pasqualepasqueflower

Relevant (possible dupe?) #26575683 – Dhobi

Answer 1

The ordering comparison operators are stricter about types in Python 3, as described here:

The ordering comparison operators (<, <=, >=, >) raise a TypeError exception when the operands don’t have a meaningful natural ordering.

Python 2 sorts None before any string (even empty string):

>>> None < None
False

>>> None < "abc"
True

>>> None < ""
True

In Python 3 any attempts at ordering NoneType instances result in an exception:

>>> None < "abc"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unorderable types: NoneType() < str()

The quickest fix I can think of is to explicitly map None instances into something orderable like "":

my_list_sortable = [(x or "") for x in my_list]

If you want to sort your data while keeping it intact, just give sort a customized key method:

def nonesorter(a):
    if not a:
        return ""
    return a

my_list.sort(key=nonesorter)

Tercentenary

Yes, I've allready noticed that. What I'm asking for is a solution. If attr was of some custom_type, I whould just override custom_type.__lt__(), but None is a built in. – Pasqualepasqueflower

It wasn't me - but one improvement would be to use "" instead of 0 as the None substitute since the OP appears to be comparing strings. – Rheumatic

But that will change the actual data inside the objects being sorted, will it not? – Pasqualepasqueflower

@AlexVhr: check the edit again for a solution that doesn't change your list. – Tercentenary

It works, but requires objects to have comparison methods, like __lt__() Otherwise it couses TypeError: unorderable types: MyObj() < MyObj() – Pasqualepasqueflower

Answer 2

For a general solution, you can define an object that compares less than any other object:

from functools import total_ordering

@total_ordering
class MinType(object):
    def __le__(self, other):
        return True

    def __eq__(self, other):
        return (self is other)

Min = MinType()

Then use a sort key that substitutes Min for any None values in the list

mylist.sort(key=lambda x: Min if x is None else x)

Betrothal

Not really that general: TypeError: unorderable types: type() < tuple() If your list contains tuples. – Liv

@neo You made a mistake in your test. Obviously using this class won't prevent other, unrelated type errors. Notice that neither of the arguments in the type error were of type MinType. – Betrothal

Ah, sorry, I had used MinType (which is a type) instead of Min. – Liv

Much better than the other solution, IMHO, and very elegant. The kind of thing I'd never have found alone. Thanks a lot ! – Marabout

Be aware that this might lead to inconsistent ordering if you have more than one of these objects competing to be the smallest. A safe way would be to have __le__ compare id(self) and id(other) when the two objects are both instances of MinType. – Vibratile

@ShioT The intent is to use the Min object as a singleton. – Betrothal

@Betrothal Hmmm.. Sorry sometimes my Java instinct still kicks in and tell me to defensively make the __init__ method throw an exception and have a getInstance class method / static method instead – Vibratile

Yeah, this is more of a "sketch" of the approach, but you could certainly make it more robust in various ways. – Betrothal

How do set an attribute this solution compares objects by? Like, I want to sort instances of a class Person by age. The standard way would be my_list.sort(key=operator.attrgetter('age')). How do I do that with this solution? – Pasqualepasqueflower

Why is the total_ordering annotation required? Can the lambda be abbreviated to key=lambda x: x or Min as suggested by an answer below? – Yucca

Answer 3

The solutions proposed here work, but this could be shortened further:

mylist.sort(key=lambda x: x or 0)

In essence, we can treat None as if it had value 0.

E.g.:

>>> mylist = [3, 1, None, None, 2, 0]
>>> mylist.sort(key=lambda x: x or 0)
>>> mylist
[None, None, 0, 1, 2, 3]

Pastis

doesn't work if you're sorting numerical lists with possible None, eg [3,1,None,2] – Micrometeorite

That puts None arbitrarily in the middle of the list if there's negatives and nonnegatives, eg [3, 1, None, None, -2, 0] – Micrometeorite

Yes. The question was about removing the error, and that removed the error. Since it did not specify what one was supposed to do with the None, I left the Nones there. – Pastis

Even if you don't care about None you should watch out for other 'falsy' values too, for instance, 0, which if you're doing a mapping to sort would be tempting to start with. – Ru

Answer 4

A general solution to deal with None independent of the types to be sorted:

my_list.sort(key=lambda x: (x is not None, x))

putting None values first.

Note that: my_list.sort(key=lambda x: (x is None, x)) puts None values last.

Nobukonoby

Answer 5

Since there are other things besides None that are not comparable to a string (ints and lists, for starters), here is a more robust solution to the general problem:

my_list.sort(key=lambda x: x if isinstance(x, str) else "")

This will let strings and any type derived from str to compare as themselves, and bin everything else with the empty string. Or substitute a different default default key if you prefer, e.g. "ZZZZ" or chr(sys.maxunicode) to make such elements sort at the end.

Rejection

how would you sort numerical lists with possible nones, (see comment to fralau) – Micrometeorite

In response to alancalvitti's 2-year old question - fralau took advantage of None being treated as False in the "or" comparison statement, which is an effective and simple way to do it. If you wish to be more explicit on numerical lists (presuming only values > 0 ): my_list.sort(key=lambda x: 0 if x is None else x) – Fouquet