How can I pickle suds results?

Question

To avoid repeatedly accessing a SOAP server during development, I'm trying to cache the results so I can run the rest of my code without querying the server each time.

With the code below I get a PicklingError: Can't pickle <class suds.sudsobject.AdvertiserSearchResponse at 0x03424060>: it's not found as suds.sudsobject.AdvertiserSearchResponse when I try to pickle a suds result. I guess this is because the classes are dynamically created.

import pickle
from suds.client import Client

client = Client(...)
result = client.service.search(...)

file = open('test_pickle.dat', 'wb')
pickle.dump(result, file, -1)
file.close()

If I drop the -1 protocol version from pickle.dump(result, file, -1), I get a different error:

TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

Is pickling the right thing to do? Can I make it work? Is there a better way?

Iene

Answer 1

As the error message you're currently getting is trying to tell you, you're trying to pickle instances that are not picklable (in the ancient legacy pickle protocol you're now using) because their class defines __slots__ but not a __getstate__ method.

However, even altering their class would not help because then you'd run into the other problem -- which you already correctly identified as being likely due to dynamically generated classes. All pickle protocols serialize classes (and functions) "by name", essentially constraining them to be at top-level names in their modules. And, serializing an instance absolutely does require serializing the class (how else could you possibly reconstruct the instance later if the class was not around?!).

So you'll need to save and reload your data in some other way, breaking your current direct dependence on concrete classes in suds.sudsobject in favor of depending on an interface (either formalized or just defined by duck typing) that can be implemented both by such concrete classes when you are in fact accessing the SOAP server, or simpler "homemade" ones when you're loading the data from a file. (The data representing instance state can no doubt be represented as a dict, so you can force it through pickle if you really want, e.g. via the copy_reg module which allows you to customize serialize/deserialize protocols for objects that you're forced to treat non-invasively [[so you can't go around adding __getstate__ or the like to their classes]] -- the problem will come only if there's a rich mesh of mutual references among such objects).

Haught

Is there a way to get the definition of a dynamically created class so that it can be saved to a file and then used as a normal class? – Gravelly

Answer 2

You are pickling the class object itself, and not instance objects of the class. This won't work if the class object is recreated. However, pickling instances of the class will work as long as the class object exists.

Firebird

I think I follow what you're saying, but having checked, type(result) is <type 'instance'> and result.__class__ is <class suds.sudsobject.AdvertiserSearchResponse at 0x03373060> so am I not trying to pickle the instance? – Iene

From the pasted code, it seems you are pickling an instance, but the error you're referring suggests you are pickling a class. – Yajairayajurveda

Okay, I'm not going completely nuts then :) I get a different error if I drop the -1 for latest protocol which I'll amend the question to include see if it drops any more light. – Iene