pickle/zodb: how to handle moving .py files with class definitions?

Question

I'm using ZODB which, as I understand it, uses pickle to store class instances. I'm doing a bit of refactoring where I want to split my models.py file into several files. However, if I do this, I don't think pickle will be able to find the class definitions, and thus won't be able to load the objects that I already have stored in the database. What's the best way to handle this problem?

Snaffle

Answer 1

You can create aliases; because one models.py modules is being split into multiple new modules you can only do this by importing your classes into the old location.

Both methods cause new copies of your instance pickles to refer to the new location; if you can force a write on all instances of the moved classes you don't need to retain the aliases. You can do this by setting _p_changed to True on your instances that you want to be written again.

So, to create the aliases, import your moved classes in the old location:

from newmodule1 import MyClass1, MyClass2
from newmodule2 import MyClass3

If you only rename a module (so the same classes all are found in one new location, could be a set of imports themselves), you can also create a sys.modules entry for the old name:

import sys
import newmodule

sys.modules['full.path.to.old.module] = newmodule

Marge

ah, this seems to be what I want. So pickle, when loading the existing instances, will look in the old models.py, find the classname, and load the class properly, but the loaded class will actually be in the new location... clever – Snaffle

@Claudiu: Exactly, so when writing a new pickle for the class, it'll use the new module path. – Marge

Answer 2

As long as you want to make the pickle loadable without performing a migration to the new class model structure: you can use alias imports of the refactored classes inside the location of the old model.py.

Elmerelmina

Answer 3

Unfortunately, there is no easy solution. You can convert your old-style objects with the refactored ones (I mean classes which are in another file/module) by the following schema

1. add the refactored classes to your code without removing the old-ones
2. walk through your DB starting from the root and replacing all old objects with new equivalents
3. compress your database (that's important)
4. now you can remove your old classes from the sources

Shaky

aye i was hoping to not have to do something like that. the alias solution seems promising... – Snaffle

@Snaffle I don't like the aliasing solution since it brings mess in code. But it works of course :-) – Shaky