pickle error assert id(obj) not in self.memo

Question

I am using dill (advanced version of pickle) right now. I want to serialize my object, but I get this error:

/usr/lib/python2.7/pickle.pyc in memoize(self, obj)
    242         if self.fast:
    243             return
--> 244         assert id(obj) not in self.memo
    245         memo_len = len(self.memo)
    246         self.write(self.put(memo_len))

Can someone tell me the possibility that made this error or how can I solved this?

Neela

Can you show us the code that throws the error? – Kingcup

At first glance then, dill is violating a pickle invariant; recursive structures should not lead to circular references being pickled again. self.memo tracks what has already been pickled before. – Possing

@Kingcup I am sorry the code is very very long and multiple files ... so I hardly show you ... but I am using Numpy, Scipy, Theano framework ... – Neela

@MartijnPieters so I should try to search the circular references that cause that problem ? – Neela

@psuedobot: perhaps; I don't know dill but the id(obj) not in self.memo assert is triggered if id(obj) has been seen before. – Possing

@MartijnPieters: I'm the dill author. It's just a big collection of using copy_reg for different objects, and I don't violate any pickle invariants… I do however do a few workarounds so all sorts of objects are added to the memo of pickle when you import dill. There's no duplication in adding to memo, unless you do something weird like import pickle, then copy_reg some stuff, then import dill -- and that's probably even fine in many cases. – Stunk

@MikeMcKerns: Right; I hadn't had time to look into dill internals at all. It sounds as if the OP should try to produce a minimal example that triggers the problem? – Possing

@MartijnPieters: absolutely. – Stunk

@MartijnPieters I am already tried to look my code and I didnt import pickle. I only import dill. – Neela

So? dill uses pickle; but follow Mike's advice. – Possing

@MartijnPieters now I already turn on the trace. And right before the assertion failed, it shown the pickle tried to serialized in line def __init___(self, ...) – Neela

@Neela It would help if you update your question to show the trace, or results of any of the detect methods you have tried as shown below in my post. It's much harder to help w/o seeing what you are seeing. – Stunk

Answer 1

Without you posting a reduced version of your code, it's hard to help. However, dill has some builtin detection methods. Look at dill.detect.

>>> # trace dill's pickling of objects, by printing out step by step trace 
>>> dill.detect.trace(True)

Or by object inspection.

>>> dill.detect.badobjects(yourfailingobject, depth=1)

There's also dill.detect.badtypes and so on.

Or you can trace down how objects relate to each other, with dill.detect.parent, dill.detect.children, dill.detect.reference, and so on.

Here's an example of using dill (plus objgraph for visualization) to track down circular references. https://github.com/uqfoundation/dill/issues/58

There's also a big list of all that dill does not know how to serialize in dill._objects -- at least the first 15 sections of the python standard library, plus some others.

Stunk

now I have tried dill.detect.trace(True). And the last row of the INFO before error pointing at the row where I coded def __init__(self, a, b, c) from the class's object that I want serialize. – Neela

@psuedobot: hard to tell what's going on from your above comment – Stunk