Is there a high performance way to replace the BinaryFormatter in .NET5?

Question

Before .NET5 we serialize/deserialize the Bytes/Object by these code:

    private static byte[] StructToBytes<T>(T t)
    {
        using (var ms = new MemoryStream())
        {
            var bf = new BinaryFormatter();
            bf.Serialize(ms, t);
            return ms.ToArray();
        }
    }

    private static T BytesToStruct<T>(byte[] bytes)
    {
        using (var memStream = new MemoryStream())
        {
            var binForm = new BinaryFormatter();
            memStream.Write(bytes, 0, bytes.Length);
            memStream.Seek(0, SeekOrigin.Begin);
            var obj = binForm.Deserialize(memStream);
            return (T)obj;
        }
    }

But the BinaryFormatter will be removed for the security reason:

https://learn.microsoft.com/en-us/dotnet/standard/serialization/binaryformatter-security-guide

So is there some simple but high performance method to replace BinaryFormatter?

Higginbotham

If not just security and performance but also high compatibility with BinaryFormatter matters you can check out my binary serializer here. See also the security notes, a size comparison and a related question. – Salangia

Answer 1

In my project, which we recently migrated from .NET Core 3.1 to .NET 5, I swapped out our BinarySerializer code with Protobuf-net: https://github.com/protobuf-net/protobuf-net

The code was almost exactly the same, and the project is very reputable with (currently) 22 million downloads and 3.2k stars on GitHub. It is very fast and has none of the security baggage surrounding BinarySerializer.

Here's my class for byte[] serialization:

public static class Binary
{
    /// <summary>
    /// Convert an object to a Byte Array, using Protobuf.
    /// </summary>
    public static byte[] ObjectToByteArray(object obj)
    {
        if (obj == null)
            return null;

        using var stream = new MemoryStream();

        Serializer.Serialize(stream, obj);

        return stream.ToArray();
    }

    /// <summary>
    /// Convert a byte array to an Object of T, using Protobuf.
    /// </summary>
    public static T ByteArrayToObject<T>(byte[] arrBytes)
    {
        using var stream = new MemoryStream();

        // Ensure that our stream is at the beginning.
        stream.Write(arrBytes, 0, arrBytes.Length);
        stream.Seek(0, SeekOrigin.Begin);

        return Serializer.Deserialize<T>(stream);
    }
}

I did have to add attributes to the class I serialized. It was decorated with [Serializable] only, and although I understand Protobuf can work with a lot of common decorations, that one didn't work. From the example on github:

[ProtoContract]
class Person {
    [ProtoMember(1)]
    public int Id {get;set;}
    [ProtoMember(2)]
    public string Name {get;set;}
    [ProtoMember(3)]
    public Address Address {get;set;}
}

[ProtoContract]
class Address {
    [ProtoMember(1)]
    public string Line1 {get;set;}
    [ProtoMember(2)]
    public string Line2 {get;set;}
}

In my case I am caching things in Redis, and it worked great.

It is also possible to re-enable this, in your .csproject file:

<PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>
</PropertyGroup>

...But it's a bad idea. BinaryFormatter is responsible for many of .NET's historical vulnerabilities, and it can't be fixed. It will likely become completely unavailable in future versions of .NET, so replacing it is the right move.

Culicid

#44799907 – Giacopo

Don't be discouraged by the "security" concerns in the abovementioned SO question. The alternatives to BinaryFormatter presented by Microsoft (XmlSerializer and System.Text.JSON) are equally vulnerable to snooping. All of which can be mitigated by encrypting the stream. The vulnerabilities of BinaryFormatter is much more serious. A malformed binary data source may cause anything from a crash, to being a vector for malware. – Entophyte

@Entophyte That was my conclusion as well. It is important to know it isn't secure out of the box, but it isn't directly vulnerable to unsolvable exploits either. – Culicid

@BrianMacKay hey man, I was copy pasting your code and noticed the part where you're deserializing, "Ensure that our stream is at the beginning". don't you think you're doing extra job here, just pass the arrBytes into MemoryStream constructor, new MemoryStream(arrbytes)... what's the problem with that? wouldn't that "Ensure that our stream is at the begining" as well? – Porfirioporgy

@AlirezaJamali What happens when you do it without that? I think I was getting an error for being at the end of the stream - it's been awhile however. – Culicid

@BrianMacKay well I managed to do it without that with no error, I was curious if doing that is necessary and if you don't, it might cause problems at future, thanks btw – Porfirioporgy

Answer 2

if you are using .NET Core 5 or greater, you can use the new System.Text.Json.JsonSerializer.Serialize and System.Text.Json.JsonSerializer.Deserialize like so:

public static class Binary
{
    /// <summary>
    /// Convert an object to a Byte Array.
    /// </summary>
    public static byte[] ObjectToByteArray(object objData)
    {
        if (objData == null)
            return default;

       return Encoding.UTF8.GetBytes(JsonSerializer.Serialize(objData, GetJsonSerializerOptions()));
    }

    /// <summary>
    /// Convert a byte array to an Object of T.
    /// </summary>
    public static T ByteArrayToObject<T>(byte[] byteArray)
    {
        if (byteArray == null || !byteArray.Any())          
            return default;
            
        return JsonSerializer.Deserialize<T>(byteArray, GetJsonSerializerOptions());
    }

    private static JsonSerializerOptions GetJsonSerializerOptions()
    {
        return new JsonSerializerOptions()
        {
            PropertyNamingPolicy = null,
            WriteIndented = true,
            AllowTrailingCommas = true,
            DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
        };
    }
}

Ejector

Answer 3

There is an option to use it in .NET Core 5:

Just add

<EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>

To the project like:

<PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>
</PropertyGroup>

I believe it will work.

Roye

Yes, I had added it, but i think it's not the best solution for the risk. – Higginbotham

It is the best way until one can change the code to use an alternative. – Sodium

@Higginbotham it's the worst way and has a time limit. .NET 5.0 is a "current" version which means it won't be supported once .NET 5 comes out next year. Don't expect this unsafe switch to work any more. Protocol Buffers are a great, cross-platform, widely supported alternative – Blacktail

there is security concern so, this is not recommected for the production – Kulak

Interestingly, we're in .net 8 (beta) now and this class is STILL supported. But @Panagiotis Kanavos's warning is still valid. Someday it WILL go away. And until then it WILL be unsafe, so use it to your own peril. – Fatherinlaw

Still exists doesn't mean still supported or that applications that use it indirectly won't crash because some functionality was removed. It's already highly unsafe from a security standpoint, which is why the type was deprecated to begin with. This is an ongoing process in .NET Core. The .NET team couldn't just yank out something that was used in so many other libraries (including WCF) so they remove functionality little by little. – Blacktail

In 2023 there's no real reason to use BinaryFormatter - gRPC is now ubiquitous, and so are the Protocol Buffers it uses. CoreWCF doesn't use BinaryFormatter but binary-endoded XML for binary bindings – Blacktail

There is zero security risk if no 3rd party is involved. If you save and load your own stuff, the security issue is not a concern. – Kumiss

Answer 4

While an old thread, it's still relevant, especially if you find yourself dealing with code storing .NET data in Memcached for example (or Redis, or secondary storage on-prem or in a cloud). BinaryFormatter has the security problems mentioned in the OP, and also has performance and size issues.

A great alternative is the MessagePack format, and more specifically the MessagePack NuGet package for .NET solutions.

It's secure, maintained, faster, and smaller all around. See the benchmarks for details.

ZeroFormatter also appears to be a great alternative.

In today's cloud-centric solutions where sizing and capacity are important for lowering costs, these are extremely helpful.

Mim

Answer 5

There is a new System.Formats.Cbor namespace for reading and writing Concise Binary Object Representation, which is a successor to MessagePack. You need to install a NuGet package in order to use it.

Crag

Answer 6

public static T Clone<T>(this T source) where T : class
        {
            if (!typeof(T).IsSerializable)
                throw new ArgumentException(string.Format("The type '{0}' must be serializable", source.GetType().Name));

            // Don't serialize a null object, simply return the default for that object
            if (ReferenceEquals(source, null))
                return default(T);

            using (Stream stream = new MemoryStream())
            {
                var writer = new BinaryWriter(stream, Encoding.UTF8, false);
                
                DataContractJsonSerializer js = new DataContractJsonSerializer(typeof(T));
                js.WriteObject(stream, source);
                stream.Seek(0, SeekOrigin.Begin);

                // Return deserialzed object
                return (T)js.ReadObject(stream);
            }
        }

Valerio

Answer 7

Totally agree gRPC and protocol buffers are the way to go. However in Net8 I recently got this message when compiling my app for publishing

DX Tools\Patch GeneratorUI.resx : warning MSB3825: Resource "ImageList1.ImageStream" of type "System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" is deserialized via BinaryFormatter at runtime.

BinaryFormatter is deprecated due to possible security risks and will be removed with .NET 9. If you wish to continue using it, set property "GenerateResourceWarnOnBinaryFormatterUse" to false. 2>DX Tools\Patch GeneratorUI.resx : warning MSB3825: More information: https://aka.ms/msbuild/net8-binaryformatter

So what this is saying is that the standard ImageList control in mscorlib uses the binary formatter. Does this mean that MS will have to replace the control (or mscorlib) for Net9 or are we being urged to replace the control with something else ?

Swords