protobuf and List<object> - how to serialize / deserialize?

Question

I have a List<object> with different types of objects in it like integers, strings, and custom types. All custom types are protobuf-adjusted. What I wanna do now is to serialize / deserialize this list with protobuf.net. Up until now I suspect that I have to declare each and every type explicitly, which is unfortunately not possible with these mixed-list constructs. Because the binary formater has no problems to do these things I hope that I missed something and that you can help me out. So my question is how to deal with objects in protobuf.net.

Unamerican

please also tell me that how much data consumption will be there to serialize a list having more than 10,000 element as an example? – Diatropism

Answer 1

(disclosure: I'm the author of protobuf-net)

BinaryFormatter is a metadata-based serializer; i.e. it sends .NET type information about every object serialized. protobuf-net is a contract-based serializer (the binary equivalent of XmlSerializer / DataContractSerializer, which will also reject this).

There is no current mechanism for transporting arbitrary objects, since the other end will have no way of knowing what you are sending; however, if you have a known set of different object types you want to send, there may be options. There is also work in the pipeline to allow runtime-extensible schemas (rather than just attributes, which are fixed at build) - but this is far from complete.

---

This isn't ideal, but it works... it should be easier when I've completed the work to support runtime schemas:

using System;
using System.Collections.Generic;
using ProtoBuf;
[ProtoContract]
[ProtoInclude(10, typeof(DataItem<int>))]
[ProtoInclude(11, typeof(DataItem<string>))]
[ProtoInclude(12, typeof(DataItem<DateTime>))]
[ProtoInclude(13, typeof(DataItem<Foo>))]
abstract class DataItem {
    public static DataItem<T> Create<T>(T value) {
        return new DataItem<T>(value);
    }
    public object Value {
        get { return ValueImpl; }
        set { ValueImpl = value; }
    }
    protected abstract object ValueImpl {get;set;}
    protected DataItem() { }
}
[ProtoContract]
sealed class DataItem<T> : DataItem {
    public DataItem() { }
    public DataItem(T value) { Value = value; }
    [ProtoMember(1)]
    public new T Value { get; set; }
    protected override object ValueImpl {
        get { return Value; }
        set { Value = (T)value; }
    }
}
[ProtoContract]
public class Foo {
    [ProtoMember(1)]
    public string Bar { get; set; }
    public override string ToString() {
        return "Foo with Bar=" + Bar;
    }
}
static class Program {
    static void Main() {
        var items = new List<DataItem>();
        items.Add(DataItem.Create(12345));
        items.Add(DataItem.Create(DateTime.Today));
        items.Add(DataItem.Create("abcde"));
        items.Add(DataItem.Create(new Foo { Bar = "Marc" }));
        items.Add(DataItem.Create(67890));

        // serialize and deserialize
        var clone = Serializer.DeepClone(items);
        foreach (DataItem item in clone) {
            Console.WriteLine(item.Value);
        }
    }
}

Gardell

Thanks Gravell, actualy I realy like your work and hoped for an answer directly from you. I know which datatypes are in the list so I'm really curious about your solution. – Unamerican

Thanks Gravell, can you give an estimate when you'll have finished the support of runtime schemas? – Unamerican

I'd rather not; I gave an estimate before, and it didn't fit... there are a lot of edge-cases to fit, even before I optimise it. If I had to guess, I reckon that unless I get a good block of free time (which I don't expect) it'll take another couple of months. I'm also trying to keep the "trunk" up to date with the latest wire-format changes, which makes things even more fun. – Gardell

Answer 2

There is a way of doing this, albeit not a very clean way, by using a wrapper object that utilises another serialisation mechanism that supports arbitrary objects. I am presenting an example below using JSON but, like I said, resorting to a different serialisation tool seems like defeating the purpose of using protobuf:

[DataContract]
public class ObjectWrapper
{
    [DataMember(Order = 1)]
    private readonly string _serialisedContent;

    [DataMember(Order = 2)]
    private readonly string _serialisedType;

    public object Content { get; private set; }

    [UsedImplicitly]
    private ObjectWrapper() { }

    public ObjectWrapper(object content)
    {
        _serialisedContent = JsonConvert.SerializeObject(content);
        _serialisedType = content.GetType().FullName;
        Content = content;
    }

    [ProtoAfterDeserialization]
    private void Initialise()
    {
        var type = Type.GetType(_serialisedType);

        Content = type != null
            ? JsonConvert.DeserializeObject(_serialisedContent, type)
            : JsonConvert.DeserializeObject(_serialisedContent);
    }
}

EDIT: This can also be done using C#'s built-in binary serialisation

[DataContract]
public class ObjectWrapper
{
    [DataMember(Order = 1)]
    private readonly string _serialisedContent;

    public object Content { get; private set; }

    [UsedImplicitly]
    private ObjectWrapper() { }

    public ObjectWrapper(object content)
    {
        using (var stream = new MemoryStream())
        {
            var formatter = new BinaryFormatter();
            formatter.Serialize(stream, content);
            stream.Flush();
            stream.Position = 0;
            _serialisedContent = Convert.ToBase64String(stream.ToArray());
        }
    }

    [ProtoAfterDeserialization]
    private void Initialise()
    {
        var data = Convert.FromBase64String(source);
        using (var stream = new MemoryStream(data))
        {
            var formatter = new BinaryFormatter();
            stream.Seek(0, SeekOrigin.Begin);
            return formatter.Deserialize(stream);
        }
    }
}

Torsk

Answer 3

List<YourClass> list;
ProtoBuf.Serializer.Deserialize<List<YourClass>>(filestream);

Gyp

Is that an extra >? – Rinaldo