How to serialize / deserialize immutable list type in c#

Question

If I have a class defined

[DataContract()]
class MyObject {
    [DataMember()]
    ImmutableList<string> Strings { get; private set}
}

The ImmutableList<T> type comes from the immutables library https://www.nuget.org/packages/Microsoft.Bcl.Immutable. Note that the class ImmutableList does not have a default constructor or a mutable Add method. Adding things to the list take the form.

myList = myList.Add("new string");

Can I add some custom support to the .NET serialization mechanism to support this type and show it how to deserialize it?

Currently the collection is just skipped on deserialization though it is fine to serialize it.

Dimension

Also, see immutability-and-xml-serialization – Checani

Answer 1

There is another clean way to do this via IDataContractSurrogate interface. The DataContractSerializer allows you to provide a surrogate for non serializable objects. Below is the example and test case for ImmutableList<T>. It uses reflection and probably could be optimized by smarter folks than me but here it is.

TESTCASE

using FluentAssertions;
using System.Collections.Immutable;
using System.IO;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Xml;
using Xunit;

namespace ReactiveUI.Ext.Spec
{
    [DataContract(Name="Node", Namespace="http://foo.com/")]
    class Node
    {
        [DataMember()]
        public string Name;
    }

    [DataContract(Name="Fixture", Namespace="http://foo.com/")]
    class FixtureType
    {
        [DataMember()]
        public ImmutableList<Node> Nodes;

        public FixtureType(){
            Nodes = ImmutableList<Node>.Empty.AddRange( new []
            { new Node(){Name="A"}
            , new Node(){Name="B"}
            , new Node(){Name="C"}
            });
        }
    }


    public class ImmutableSurrogateSpec
    {  
        public static string ToXML(object obj)
            {
                var settings = new XmlWriterSettings { Indent = true };

                using (MemoryStream memoryStream = new MemoryStream())
                using (StreamReader reader = new StreamReader(memoryStream))
                using (XmlWriter writer = XmlWriter.Create(memoryStream, settings))
                {
                    DataContractSerializer serializer =
                      new DataContractSerializer
                          ( obj.GetType()
                          , new DataContractSerializerSettings() { DataContractSurrogate = new ImmutableSurrogateSerializer() }
                          );
                    serializer.WriteObject(writer, obj);
                    writer.Flush();
                    memoryStream.Position = 0;
                    return reader.ReadToEnd();
                }
            }

        public static T Load<T>(Stream data)
        {
            DataContractSerializer ser = new DataContractSerializer
                  ( typeof(T)
                  , new DataContractSerializerSettings() { DataContractSurrogate = new ImmutableSurrogateSerializer() }
                  );
            return (T)ser.ReadObject(data);
        }

        public static T Load<T>(string data)
        {
            using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(data)))
            {
                return Load<T>(stream);
            }
        }

        [Fact]
        public void ShouldWork()
        {
            var o = new FixtureType();

            var s = ToXML(o);

            var oo = Load<FixtureType>(s);

            oo.Nodes.Count().Should().Be(3);
            var names = oo.Nodes.Select(n => n.Name).ToList();
            names.ShouldAllBeEquivalentTo(new[]{"A", "B", "C"});

        }

    }
}

IMPLEMENTATION

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using System.Runtime.Serialization;

namespace ReactiveUI.Ext
{
    class ImmutableListListConverter<T>
    {
        public static ImmutableList<T> ToImmutable( List<T> list )
        {
            return ImmutableList<T>.Empty.AddRange(list);
        }

        public static List<T> ToList(ImmutableList<T> list){
            return list.ToList();
        }

        public static object ToImmutable( object list )
        {
            return ToImmutable(( List<T> ) list);
        }

        public static object ToList(object list){
            return ToList(( ImmutableList<T> ) list);
        }

    }

    static class ImmutableListListConverter {


        static ConcurrentDictionary<Tuple<string, Type>, Func<object,object>> _MethodCache 
            = new ConcurrentDictionary<Tuple<string, Type>, Func<object,object>>();

        public static Func<object,object> CreateMethod( string name, Type genericType )
        {
            var key = Tuple.Create(name, genericType);
            if ( !_MethodCache.ContainsKey(key) )
            {
                _MethodCache[key] = typeof(ImmutableListListConverter<>)
                    .MakeGenericType(new []{genericType})
                    .GetMethod(name, new []{typeof(object)})
                    .MakeLambda();
            }
            return _MethodCache[key];
        }
        public static Func<object,object> ToImmutableMethod( Type targetType )
        {
            return ImmutableListListConverter.CreateMethod("ToImmutable", targetType.GenericTypeArguments[0]);
        }

        public static Func<object,object> ToListMethod( Type targetType )
        {
            return ImmutableListListConverter.CreateMethod("ToList", targetType.GenericTypeArguments[0]);
        }

        private static Func<object,object> MakeLambda(this MethodInfo method )
        {
            return (Func<object,object>) method.CreateDelegate(Expression.GetDelegateType(
            (from parameter in method.GetParameters() select parameter.ParameterType)
            .Concat(new[] { method.ReturnType })
            .ToArray()));
        }

    }

    public class ImmutableSurrogateSerializer : IDataContractSurrogate
    {
        static ConcurrentDictionary<Type, Type> _TypeCache = new ConcurrentDictionary<Type, Type>();

        public Type GetDataContractType( Type targetType )
        {
            if ( _TypeCache.ContainsKey(targetType) )
            {
                return _TypeCache[targetType];
            }

            if(targetType.IsGenericType && targetType.GetGenericTypeDefinition() == typeof(ImmutableList<>)) 
            {
                return _TypeCache[targetType] 
                    = typeof(List<>).MakeGenericType(targetType.GetGenericArguments());
            }
            else
            {
                return targetType;
            }
        }

        public object GetDeserializedObject( object obj, Type targetType )
        {
            if ( _TypeCache.ContainsKey(targetType) )
            {
               return ImmutableListListConverter.ToImmutableMethod(targetType)(obj);
            }
            return obj;
        }

        public object GetObjectToSerialize( object obj, Type targetType )
        {
            if ( targetType.IsGenericType && targetType.GetGenericTypeDefinition() == typeof(ImmutableList<>) )
            {
               return ImmutableListListConverter.ToListMethod(targetType)(obj);
            }
            return obj;
        }

        public object GetCustomDataToExport( Type clrType, Type dataContractType )
        {
            throw new NotImplementedException();
        }

        public object GetCustomDataToExport( System.Reflection.MemberInfo memberInfo, Type dataContractType )
        {
            throw new NotImplementedException();
        }


        public void GetKnownCustomDataTypes( System.Collections.ObjectModel.Collection<Type> customDataTypes )
        {
            throw new NotImplementedException();
        }


        public Type GetReferencedTypeOnImport( string typeName, string typeNamespace, object customData )
        {
            throw new NotImplementedException();
        }

        public System.CodeDom.CodeTypeDeclaration ProcessImportedType( System.CodeDom.CodeTypeDeclaration typeDeclaration, System.CodeDom.CodeCompileUnit compileUnit )
        {
            throw new NotImplementedException();
        }

        public ImmutableSurrogateSerializer() { }
    }
}

Dimension

It's quite easy to prove it is working by commenting out the line that adds the surrogate to the serializer and watch the test fail. – Dimension

Note I've stopped using Datacontract Serializer and now use the json.net serializers which "just work" for immutable collections. – Dimension

Bit late, but are you sure GetObjectToSerializer shouldn't take both the actual type and the target type into account? I had to adjust that method to get the an ImmutableSurrogate to work with an ImmutableHashSet. – Slattery

Based on this answer i've implemented the ISerializationSurrogateProvider interface for the .NET Standard library. It's basically the same as the IDataContractSurrogate interface for .Net 4.5. It should also perform a little bit better because we avoid coping the entire collection for serialization. github.com/bp74/System.Collections.Immutable.Surrogate – Galvanize

Answer 2

Heh; I can imagine what is happening here... the generated code is probably doing (paraphrasing):

var list = obj.Strings;
while(CanReadNextItem()) {
    list.Add(ReadNextItem());
}

The problem is that the BCL immutable API would require you to catch the result each time, i.e.

var list = obj.Strings;
while(CanReadNextItem()) {
    list = list.Add(ReadNextItem());
}
obj.Strings = list; // the private set is not a problem for this

Pre-existing list deserialization code doesn't work this way because it has never needed to - and indeed, there are many different implementations of Add, some of which return non-void results which are required to be ignored.

The lack of a non-public constructor may also upset it a bit, but if this was the main problem, I would kinda expect an exception when it tries to create a non-null list.

Of course, in performance terms, the list = list.Add(...) API probably isn't the most appropriate one to use anyway (although it should work).

I blogged on this topic recently (in the context of protobuf-net, which has now been updated to work with these collection types): http://marcgravell.blogspot.co.uk/2013/09/fun-with-immutable-collections.html Hopefully this blog article should explain why the differences mean that it doesn't play nicely with existing serialization techniques, and how serialization libraries might be updated to work with this scenario.

To answer the question directly, I would say the answer is simply: because the required changes to support immutable collections have not yet been made to DataContractSerializer. I have no knowledge of whether there is a plan to address this. But: I happily declare: "works in protobuf-net" ;p

Roseliaroselin

The BCL libs have the Builder pattern where you can, within a transaction, go mutable. That would be the correct thing to do but there is no hook within DataContractSerializer to add this in myself :( ? – Dimension

@Dimension yes, I discuss that (the builder API) fully in the blog. Frankly, I would be surprised if any serialization libraries had an extension point that was directly applicable to this situation. The changes I had to make to protobuf-net were beyond what could be achieved via surrogates, etc. And the fact that the builder API depends on knowledge of the non-generic utility class matching the generic concrete list means that it can't reasonably be expected to just work in serializers that didn't already know about this pattern. – Roseliaroselin

My solution is to use a hidden private field as the actual DataMember and using the OnSerializing and OnDeserialized hooks proxy the immutable collection through the mutable list. It works and from a public interface perspective the collection it is still immutable. – Dimension

@Dimension it makes a mess of the object, though; as per my other comment - I'd prefer to see a dedicated DTO model in that scenario – Roseliaroselin

I agree. I've just found a hook for datacontract serializer surrogates msdn.microsoft.com/en-us/library/… which is the official way to implement the hack pattern I did manually. Not sure I can get it to work for generics but I will give it a go. – Dimension

I got it working with the surrogate serializer. It works very well. I'll add another answer as it is significantly different to the previous method. – Dimension

Answer 3

One way to do this is to use a proxy mutable list and use the OnSerializing and OnDeserialized hooks

[DataContract()]
class MyObject {

    public ImmutableList<string> Strings { get; private set}

    [DataMember(Name="Strings")]
    private List<String> _Strings;

    [OnSerializing()]
    public void OnSerializing(StreamingContext ctxt){
        _Strings = Strings.ToList();
    }

    [OnDeserialized()]
    public void OnDeserialized(StreamingContext ctxt){
        Strings = ImmutableList<string>.Empty.AddRange(_Strings);
    }
}

It's not super pretty but as Marc Gravell noted in his answer, DataContract serializer is broken with respects to immutable collections and there are no simple hooks to teach it how to behave without the above type of hack.

UPDATE

DataContract serializer is not broken. There is a way to hook surrogates in. See this separate answer showing an alternate technique.

https://mcmap.net/q/1433578/-how-to-serialize-deserialize-immutable-list-type-in-c

Dimension

As a side note - my usual guidance is: if ever serialization steps outside of what can be done trivially with the chosen serializer, then rather than fighting it, create a separate DTO model (MyObjectDTO, for exmaple) that does play nicely with the serializer, and add conversion methods between the two models. So MyObjectDTO would have a simple List<string>, MyObject keeps the ImmutableList<string>, and the conversion code has the magic to switch between them. – Roseliaroselin

Good advice but there is an API to handle this. IDataContractSurrogate. I just need to figure out how to write the code that maps ImmutableList<T> to List<T> and it should just work. – Dimension