Serializing a suds object in python

Question

Ok I'm working on getting better with python, so I'm not sure this is the right way to go about what I'm doing to begin with, but here's my current problem...

I need to get some information via a SOAP method, and only use part of the information now but store the entire result for future uses (we need to use the service as little as possible). Looking up the best way to access the service I figured suds was the way to go, and it was simple and worked like a charm to get the data. But now I want to save the result somehow, preferably serialized / in a database so I can pull it out later and use it the same.

What's the best way to do this, it looks like pickle/json isn't an option? Thanks!

Update Reading the top answer at How can I pickle suds results? gives me a better idea of why this isn't an option, I guess I'm stuck recreating a basic object w/ the information I need?

Occult

Answer 1

Yep, I confirm the explanation I gave in the answer you refer to -- dynamically generated classes are not easily picklable (nor otherwise easily serializable), you need to extract all the state information, pickle that state, and reconstruct the tricky sudsobject on retrieval if you really insist on using it;-).

Krugersdorp

I'm using the 'retxml=True' option to just get the raw result back and then turning that into a dict (which should be serializable) now - looks like it will work... thanks! – Occult

@pssdbt, excellent -- or you could serialize the XML itself, but I believe that unpickling a pickled dict will be faster than parsing the XML, so I think you've made the right choice. – Krugersdorp

Answer 2

I have been using following approach to convert Suds object into JSON:

from suds.sudsobject import asdict

def recursive_asdict(d):
    """Convert Suds object into serializable format."""
    out = {}
    for k, v in asdict(d).items():
        if hasattr(v, '__keylist__'):
            out[k] = recursive_asdict(v)
        elif isinstance(v, list):
            out[k] = []
            for item in v:
                if hasattr(item, '__keylist__'):
                    out[k].append(recursive_asdict(item))
                else:
                    out[k].append(item)
        else:
            out[k] = v
    return out

def suds_to_json(data):
    return json.dumps(recursive_asdict(data))

Nobby

Note that the function doesn't work with a list of suds objects. This can be resolved with [recursive_asdict(d) for d in ds]. – Adaxial

@Adaxial It does work because of the elif isinstance(v, list) check. – Micamicaela

Using this solution I'm getting the error: AttributeError: 'dict' object has no attribute 'iteritems' – Stung

@GuilhermeMatheus Hehe, this code was for Python2. I hope it's fixed now :) – Nobby

@Micamicaela I was able to parse the data from a XML response into JSON object, using the parameter retxml=True. But thank you very much for your faster edited reply! – Stung

Works great 10 years later. Thank you for the effort. – Watery

Answer 3

Here is what I came up with before researching and finding this answer. This actually works well for me on complex suds responses and also on other objects such as __builtins__ since the solution is suds agnostic:

import datetime

def object_to_dict(obj):
    if isinstance(obj, (str, unicode, bool, int, long, float, datetime.datetime, datetime.date, datetime.time)):
        return obj
    data_dict = {}
    try:
        all_keys = obj.__dict__.keys()  # vars(obj).keys()
    except AttributeError:
        return obj
    fields = [k for k in all_keys if not k.startswith('_')]
    for field in fields:
        val = getattr(obj, field)
        if isinstance(val, (list, tuple)):
            data_dict[field] = []
            for item in val:
                data_dict[field].append(object_to_dict(item))
        else:
            data_dict[field] = object_to_dict(val)
    return data_dict

This solution works and is actually faster. It also works on objects that don't have the __keylist__ attribute.

I ran a benchmark 100 times on a complex suds output object, this solutions run time was 0.04 to .052 seconds (0.045724287 average). While recursive_asdict solution above ran in .082 to 0.102 seconds so nearly double (0.0829765582 average).

I then went back to the drawing board and re-did the function to get more performance out of it, and it does not need the datetime import. I leveraged in using the __keylist__ attribute, so this will not work on other objects such as __builtins__ but works nicely for suds object output:

def fastest_object_to_dict(obj):
    if not hasattr(obj, '__keylist__'):
        return obj
    data = {}
    fields = obj.__keylist__
    for field in fields:
        val = getattr(obj, field)
        if isinstance(val, list):  # tuple not used
            data[field] = []
            for item in val:
                data[field].append(fastest_object_to_dict(item))
        else:
            data[field] = fastest_object_to_dict(val)
    return data

The run time was 0.18 - 0.033 seconds (0.0260889721 average), so nearly 4x as faster than the recursive_asdict solution.

Anthracnose

Answer 4

I made an implementation of a dummy class for Object intance of suds, and then being able to serialize. The FakeSudsInstance behaves like an original Suds Object instance, see below:

from suds.sudsobject import Object as SudsObject

class FakeSudsNode(SudsObject):

    def __init__(self, data):
        SudsObject.__init__(self)
        self.__keylist__ = data.keys()
        for key, value in data.items():
            if isinstance(value, dict):
                setattr(self, key, FakeSudsNode(value))
            elif isinstance(value, list):
                l = []
                for v in value:
                    if isinstance(v, list) or isinstance(v, dict):
                        l.append(FakeSudsNode(v))
                    else:
                        l.append(v)
                setattr(self, key, l)
            else:
                setattr(self, key, value)


class FakeSudsInstance(SudsObject):

    def __init__(self, data):
        SudsObject.__init__(self)
        self.__keylist__ = data.keys()
        for key, value in data.items():
            if isinstance(value, dict):
                setattr(self, key, FakeSudsNode(value))
            else:
                setattr(self, key, value)

    @classmethod
    def build_instance(cls, instance):
        suds_data = {}
        def node_to_dict(node, node_data):
            if hasattr(node, '__keylist__'):
                keys = node.__keylist__
                for key in keys:
                    if isinstance(node[key], list):
                        lkey = key.replace('[]', '')
                        node_data[lkey] = node_to_dict(node[key], [])
                    elif hasattr(node[key], '__keylist__'):
                        node_data[key] = node_to_dict(node[key], {})
                    else:
                        if isinstance(node_data, list):
                            node_data.append(node[key])
                        else:
                            node_data[key] = node[key]
                return node_data
            else:
                if isinstance(node, list):
                    for lnode in node:
                        node_data.append(node_to_dict(lnode, {}))
                    return node_data
                else:
                    return node
        node_to_dict(instance, suds_data)
        return cls(suds_data)

Now, after a suds call, for example below:

# Now, after a suds call, for example below
>>> import cPickle as pickle
>>> suds_intance = client.service.SomeCall(account, param)
>>> fake_suds = FakeSudsInstance.build_instance(suds_intance)
>>> dumped = pickle.dumps(fake_suds)
>>> loaded = pickle.loads(dumped)

I hope it helps.

Bondie

node_data.append(node_to_dict(lnode, {})) is wrong since it passes a dict into node_to_dict. – Inhalation

return node_data inside of if hasattr(node, '__keylist__'): should also return FakeSudsNode(node_data) since its composing a dict – Inhalation

Answer 5

The solutions suggesed above lose valuable information about class names - it can be of value in some libraries like DFP client https://github.com/googleads/googleads-python-lib where entity types might be encoded in dynamically generated class names (i.e. TemplateCreative/ImageCreative)

Here's the solution I used that preserves class names and restores dict-serialized objects without data loss (except suds.sax.text.Text which would be converted into regular unicode objects and maybe some other types I haven't run into)

from suds.sudsobject import asdict, Factory as SudsFactory

def suds2dict(d):                                                               
    """                                                                         
    Suds object serializer 
    Borrowed from https://mcmap.net/q/611193/-serializing-a-suds-object-in-python/15678861#15678861                                                     
    """                                                                         
    out = {'__class__': d.__class__.__name__}                                   
    for k, v in asdict(d).iteritems():                                          
        if hasattr(v, '__keylist__'):                                           
            out[k] = suds2dict(v)                                               
        elif isinstance(v, list):                                               
            out[k] = []                                                         
            for item in v:                                                      
                if hasattr(item, '__keylist__'):                                
                    out[k].append(suds2dict(item))                              
                else:                                                           
                    out[k].append(item)                                         
        else:                                                                   
            out[k] = v                                                          
    return out                                                                  


def dict2suds(d):                                                               
    """                                                                         
    Suds object deserializer                                                    
    """                                                                         
    out = {}                                                                    
    for k, v in d.iteritems():                                                  
        if isinstance(v, dict):                                                 
            out[k] = dict2suds(v)                                               
        elif isinstance(v, list):                                               
            out[k] = []                                                         
            for item in v:                                                      
                if isinstance(item, dict):                                      
                    out[k].append(dict2suds(item))                              
                else:                                                           
                    out[k].append(item)                                         
        else:                                                                   
            out[k] = v                                                          
    return SudsFactory.object(out.pop('__class__'), out)  

Butta

Just to point out that for Python 3, replace iteritems() with items(). Thanks for the awesome answer! – Toed

I needed only the dict2suds part and had an error in the out.pop('__class__') part but removing it and replacing directly with .object(dict=out) fixed it for my case. Thank you for the tip! – Deidradeidre

Answer 6

I updated the recursive_asdict example above to be compatible with python3 (items instead of iteritems).

from suds.sudsobject import asdict
from suds.sax.text import Text

def recursive_asdict(d):
    """
    Recursively convert Suds object into dict.
    We convert the keys to lowercase, and convert sax.Text
    instances to Unicode.

    Taken from:
    https://mcmap.net/q/611193/-serializing-a-suds-object-in-python

    Let's create a suds object from scratch with some lists and stuff
    >>> from suds.sudsobject import Object as SudsObject
    >>> sudsobject = SudsObject()
    >>> sudsobject.Title = "My title"
    >>> sudsobject.JustAList = [1, 2, 3]
    >>> sudsobject.Child = SudsObject()
    >>> sudsobject.Child.Title = "Child title"
    >>> sudsobject.Child.AnotherList = ["4", "5", "6"]
    >>> childobject = SudsObject()
    >>> childobject.Title = "Another child title"
    >>> sudsobject.Child.SudObjectList = [childobject]

    Now see if this works:
    >>> result = recursive_asdict(sudsobject)
    >>> result['title']
    'My title'
    >>> result['child']['anotherlist']
    ['4', '5', '6']
   """
    out = {}
    for k, v in asdict(d).items():
        k = k.lower()
        if hasattr(v, '__keylist__'):
            out[k] = recursive_asdict(v)
        elif isinstance(v, list):
            out[k] = []
            for item in v:
                if hasattr(item, '__keylist__'):
                    out[k].append(recursive_asdict(item))
                else:
                    out[k].append(
                        item.title() if isinstance(item, Text) else item)
        else:
            out[k] = v.title() if isinstance(v, Text) else v
    return out

Bowhead

Answer 7

I like this way. We don't do the iteration ourselves, it is python that iterates when converting it to string

class Ob:
    def __init__(self, J) -> None:
        self.J = J
    
    def __str__(self):           

        if hasattr(self.J, "__keylist__"):
            self.J = {key: Ob(value) for key, value in dict(self.J).items()}
        if hasattr(self.J, "append"):
            self.J = [Ob(data) for data in sefl.J]
        return str(self.J)

result = Ob(result_soap)

Mycenaean