Why can't dill/pickle class definition?

Question

dill is a great tool for pickling most the Python objects, I use it in IPython parallel to serialize calculations. One issue I keep getting into is around dill-ing class definitions. One of the errors I get is explained below.

While trying to serialize class definitions, I keep getting AssertionError from dill. I wonder why one of these works and the other fails:

class MyClassEmpty(object):
    pass

class MyClassInit(object):
    def __init__(self):
        super(MyClassInit).__init__()

dill.dumps(MyClassEmpty)  # returns: '\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassEmptyq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNutq\x0eRq\x0f.'

dill.dumps(MyClassInit)  # AssertionError at line 244 of MyClassEmpty (assert id(obj) not in self.memo)

I'm on Python 2.7.6 using dill 0.2.2.

Angelynanger

Have you read e.g. https://mcmap.net/q/868041/-pickling-a-class-definition? Also, your super call seems incorrect - that should be super(MyClassInit, self).__init__(). – Reverend

I can execute it, there's no AssertionError thrown. – Feune

Thanks jonrsharpe! The issue was simply missing , self. Having corrected dumps() just works as expected, as in the linked post as well. – Angelynanger

@SmCaterpillar, interesting! Are you on the same Python and dill version? – Angelynanger

Hmm I used python 2.7.9, but within an IPython Notebook environment. Still that should not change anything?! – Feune

Just updated the description, I'm on 2.7.6 which can make a difference. Have seen similar discrepancy with even minor changes in the interpreter. Thanks for noting! – Angelynanger

Answer 1

I'm the author of dill. The super issue should be resolved -- see: https://github.com/uqfoundation/dill/issues/26

>>> class MyClassEmpty(object):
...   pass
... 
>>> class MyClassInit(object):
...   def __init__(self):
...     super(MyClassInit).__init__()
... 
>>> import dill
>>> 
>>> dill.dumps(MyClassEmpty)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassEmptyq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNutq\x0eRq\x0f.'
>>> dill.dumps(MyClassInit) 
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0bMyClassInitq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNU\x08__init__q\x0ecdill.dill\n_create_function\nq\x0f(cdill.dill\n_unmarshal\nq\x10U\x90c\x01\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00C\x00\x00\x00s\x14\x00\x00\x00t\x00\x00t\x01\x00\x83\x01\x00j\x02\x00\x83\x00\x00\x01d\x00\x00S(\x01\x00\x00\x00N(\x03\x00\x00\x00t\x05\x00\x00\x00supert\x0b\x00\x00\x00MyClassInitt\x08\x00\x00\x00__init__(\x01\x00\x00\x00t\x04\x00\x00\x00self(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>R\x02\x00\x00\x00\x02\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x11\x85q\x12Rq\x13c__builtin__\n__main__\nh\x0eNN}q\x14tq\x15Rq\x16utq\x17Rq\x18.'
>>> 
>>> class MyClassInit2(object):
...   def __init__(self):
...     super(MyClassInit, self).__init__()
... 
>>> dill.dumps(MyClassInit2)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x0cMyClassInit2q\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\n__module__q\x0bU\x08__main__q\x0cU\x07__doc__q\rNU\x08__init__q\x0ecdill.dill\n_create_function\nq\x0f(cdill.dill\n_unmarshal\nq\x10U\x93c\x01\x00\x00\x00\x01\x00\x00\x00\x03\x00\x00\x00C\x00\x00\x00s\x17\x00\x00\x00t\x00\x00t\x01\x00|\x00\x00\x83\x02\x00j\x02\x00\x83\x00\x00\x01d\x00\x00S(\x01\x00\x00\x00N(\x03\x00\x00\x00t\x05\x00\x00\x00supert\x0b\x00\x00\x00MyClassInitt\x08\x00\x00\x00__init__(\x01\x00\x00\x00t\x04\x00\x00\x00self(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>R\x02\x00\x00\x00\x02\x00\x00\x00s\x02\x00\x00\x00\x00\x01q\x11\x85q\x12Rq\x13c__builtin__\n__main__\nh\x0eNN}q\x14tq\x15Rq\x16utq\x17Rq\x18.'

BTW: dill punts to pickle for certain cases involving classes (primarily), and pickle throws an AssertionError as one of the 3-4 errors that it might evoke. Why there's not just a PicklingError I don't know… that might be more preferable.

Oloughlin

Thanks Mark. I appreciate you posting questions / bug reports / etc. – Oloughlin