Python: Can't pickle type X, attribute lookup failed

Question

I am trying to pickle a namedtuple:

from collections import namedtuple
import cPickle

class Foo:

    Bar = namedtuple('Bar', ['x', 'y'])

    def baz(self):
        s = set()
        s.add(Foo.Bar(x=2, y=3))
        print cPickle.dumps(s)

if __name__ == '__main__':
    f = Foo()
    f.baz()

This produces the following output:

Traceback (most recent call last):
  File "scratch.py", line 15, in <module>
    f.baz()
  File "scratch.py", line 11, in baz
    print cPickle.dumps(s)
cPickle.PicklingError: Can't pickle <class '__main__.Bar'>: attribute lookup __main__.Bar failed

What am I doing wrong? Is the problem that Bar is a member of Foo? (Moving the definition of Bar to the top level solves the problem, although I'm still curious why this happens.)

Trip

Using python3 and pickle protocol 4 fixes this – Time

Is @DaveButler 's answer correct? Can anyone confirm or deny? – Pratique

DaveButler's comment is not correct. But there is a Python-3-specific solution: see my answer. – Secede

Answer 1

Yes, the fact that it's a class member is a problem:

>>> class Foo():
...     Bar = namedtuple('Bar', ['x','y'])
...     def baz(self):
...         b = Foo.Bar(x=2, y=3)
...         print(type(b))
...
>>> a = Foo()
>>> a.baz()
<class '__main__.Bar'>

The problem is that when namedtuple() returns a type object, it isn't aware of the fact that it's being assigned to a class member - and thus, it tells the type object that its type name should be __main__.Bar, even though it should really be __main__.Foo.Bar.

Wiltz

namedtuple just doesn't play nice with classes. You might be able to write a custom __getstate__ for the object which would take care of it manually. – Wiltz

Would an early bind grab that? – Oneself

How could I use getstate? I don't know how to implement that. At the moment I'm working with a temporary solution that transfers a list of named tuples to a dictionary. – Hein

Answer 2

Nesting classes makes pickle fail, since it relies on the path of the object inside your application to reconstruct it later.

The immediate solution is to not nest classes, i.e. move Bar definition to outside Foo. Code will work all the same.

But a better thing to do is to not use pickle at all to store data. Use some other serialization format, like json, or a database, like sqlite3.

You have just hit one of the many inconveniences of pickle, if you change your code, move things around, or sometimes make small structural changes, your data becomes unloadable.

Besides that, pickle has other disadvantages: It is slow, unsecure, python-only...

Linzer

Answer 3

Using dill in place of pickle here will allow this to work

Thankyou

import dill; dill.dumps(x) – Thankyou

Also this doesn't work for me. I get: Can't pickle <class 'RoadAttributesPartition'>: it's not found as main.RoadAttributesPartition – Myeshamyhre

Caution: this works because dill pickles namedtuple classes as code that constructs a new namedtuple class. That means that tests like isinstance(x, Foo.Bar) will return False, and even unpickling the same dill-produced buffer twice will get you instances of different classes. – Secede

Answer 4

The solution here is to move your named tuple definition to the module level, then pickle works. A detailed answer is provided here:

How to pickle a namedtuple instance correctly

Heredity

Answer 5

If you came here via search engine, the problem could also be that the name of the type and the first parameter are not identical. For example:

FooBar = namedtuple('foo_bar', ['x', 'y'])

should be

FooBar = namedtuple('FooBar', ['x', 'y'])

Reade

Answer 6

In Python 3.5+, you can fix this by using typing.NamedTuple instead of collections.namedtuple:

from typing import NamedTuple

class Foo:
    class Bar(NamedTuple):
        x: int
        y: int

if __name__ == '__main__':
    import pickle
    s = {Foo.Bar(x=2, y=3)}
    p = pickle.dumps(s, 4)
    assert pickle.loads(p) == s

Secede