Can Json.NET serialize / deserialize to / from a stream?

Question

I have heard that Json.NET is faster than DataContractJsonSerializer, and wanted to give it a try...

But I couldn't find any methods on JsonConvert that take a stream rather than a string.

For deserializing a file containing JSON on WinPhone, for example, I use the following code to read the file contents into a string, and then deserialize into JSON. It appears to be about 4 times slower in my (very ad-hoc) testing than using DataContractJsonSerializer to deserialize straight from the stream...

// DCJS
DataContractJsonSerializer dc = new DataContractJsonSerializer(typeof(Constants));
Constants constants = (Constants)dc.ReadObject(stream);

// JSON.NET
string json = new StreamReader(stream).ReadToEnd();
Constants constants = JsonConvert.DeserializeObject<Constants>(json);

Am I doing it wrong?

Merrilee

Answer 1

UPDATE: This no longer works in the current version, see below for correct answer (no need to vote down, this is correct on older versions).

Use the JsonTextReader class with a StreamReader or use the JsonSerializer overload that takes a StreamReader directly:

var serializer = new JsonSerializer();
serializer.Deserialize(streamReader);

Unbeknown

Pretty sure this no longer works. You have to use a JsonReader or TextReader – Chemotaxis

You may want to include the version number this is still working on so people know when to scroll down. – Nakada

@Chemotaxis yup JsonTextReader(givenStreamReader) is the way to go now – Overstuff

Thank you for taking the time to edit your answer re it's working status and answer recommendation – Nipissing

"no need to vote down, this is correct on older versions" - It's no longer relevant to most people, so it should be voted down as a better answer exists. – Bethesda

Answer 2

The current version of Json.net does not allow you to use the accepted answer code. A current alternative is:

public static object DeserializeFromStream(Stream stream)
{
    var serializer = new JsonSerializer();

    using (var sr = new StreamReader(stream))
    using (var jsonTextReader = new JsonTextReader(sr))
    {
        return serializer.Deserialize(jsonTextReader);
    }
}

Documentation: Deserialize JSON from a file stream

Leveille

JsonTextReader will close its StreamReader by default, so this example could be simplified a bit by constructing the StreamReader in the call to the JsonTextReader constructor. – Vertical

Actually, I have an OutOfMemory exception and I already use this code, pretty much exactly. Which, I believe, goes to say, this is not a guarantee - if the deserialized object is large enough, and you're stuck in a 32-bit process, you may still get memory errors with this code – Floreated

i am getting an error "The type or namespace name 'JsonTextReader' could not be found" ...any suggestions? – Caul

I needed to add stream.Position = 0; to correctly deserialize my json. – Caught

The documentation in the link does not mention anything about JsonTextReader – Kiker

If you want to control the serializer settings, use JsonSerializer.Create(); instead of the constructor. – Cernuous

@James is there a version where stream is read with async? – Barcot

Answer 3

public static void Serialize(object value, Stream s)
{
    using (StreamWriter writer = new StreamWriter(s))
    using (JsonTextWriter jsonWriter = new JsonTextWriter(writer))
    {
        JsonSerializer ser = new JsonSerializer();
        ser.Serialize(jsonWriter, value);
        jsonWriter.Flush();
    }
}

public static T Deserialize<T>(Stream s)
{
    using (StreamReader reader = new StreamReader(s))
    using (JsonTextReader jsonReader = new JsonTextReader(reader))
    {
        JsonSerializer ser = new JsonSerializer();
        return ser.Deserialize<T>(jsonReader);
    }
}

Annual

Thanks! This helped me avoid an OutOfMemoryException I was getting when I was serializing a very large object collection to a string, and then writing that string into my stream (instead of just serializing directly to the stream). – Skep

Why flush? Doesn't the Dispose call caused by the using block already do that? – Hath

Side note, because it might help others: if you use JsonSerializer ser = JsonSerializer.Create(settings); you can define which settings to use during de/serialization. – Apoplectic

One potential issue with this Serialize implementation is that it closes the Stream passed as an argument, which depending on the application can be a problem. With .NET 4.5+ you can avoid this problem by using a StreamWriter constructor overload with a parameter leaveOpen that lets you leave the stream open. – Sparke

@ŞafakGür It may not. Seen that happening to many times, so as a reflex I just flush, like the sample. – Aurelioaurelius

Answer 4

I've written an extension class to help me deserializing from JSON sources (string, stream, file).

public static class JsonHelpers
{
    public static T CreateFromJsonStream<T>(this Stream stream)
    {
        JsonSerializer serializer = new JsonSerializer();
        T data;
        using (StreamReader streamReader = new StreamReader(stream))
        {
            data = (T)serializer.Deserialize(streamReader, typeof(T));
        }
        return data;
    }

    public static T CreateFromJsonString<T>(this String json)
    {
        T data;
        using (MemoryStream stream = new MemoryStream(System.Text.Encoding.Default.GetBytes(json)))
        {
            data = CreateFromJsonStream<T>(stream);
        }
        return data;
    }

    public static T CreateFromJsonFile<T>(this String fileName)
    {
        T data;
        using (FileStream fileStream = new FileStream(fileName, FileMode.Open))
        {
            data = CreateFromJsonStream<T>(fileStream);
        }
        return data;
    }
}

Deserializing is now as easy as writing:

MyType obj1 = aStream.CreateFromJsonStream<MyType>();
MyType obj2 = "{\"key\":\"value\"}".CreateFromJsonString<MyType>();
MyType obj3 = "data.json".CreateFromJsonFile<MyType>();

Hope it will help someone else.

Overshoot

Against: it will pollute all strings with the extension methods. Workarounds: Only declare Using SomeJsonHelpersNamespace where needed or remove the this keyword and use JsonHelpers.CreateFromJsonString(someJsonString) Pro: it's so easier to use :) – Birkett

Although it could be seen as "polluting", almost half the extensions in String object could be seen the same way. This extends an object in a manner seen as useful to anyone that would consistently change from string(json) to JSON. – Gebelein

Also using Encoding.Default is bad as it will behave differently on different machines (see the big warning at the Microsoft docu). JSON is expected to be UTF-8 and this is what JsonSerializer expects. Thus it should be Encoding.UTF8. The code as is will produce garbled strings or fail to deserialize if non-ASCII characters are used. – Scorpaenid

Answer 5

I arrived at this question looking for a way to stream an open ended list of objects onto a System.IO.Stream and read them off the other end, without buffering the entire list before sending. (Specifically I'm streaming persisted objects from MongoDB over Web API.)

@Paul Tyng and @Rivers did an excellent job answering the original question, and I used their answers to build a proof of concept for my problem. I decided to post my test console app here in case anyone else is facing the same issue.

using System;
using System.Diagnostics;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace TestJsonStream {
    class Program {
        static void Main(string[] args) {
            using(var writeStream = new AnonymousPipeServerStream(PipeDirection.Out, HandleInheritability.None)) {
                string pipeHandle = writeStream.GetClientHandleAsString();
                var writeTask = Task.Run(() => {
                    using(var sw = new StreamWriter(writeStream))
                    using(var writer = new JsonTextWriter(sw)) {
                        var ser = new JsonSerializer();
                        writer.WriteStartArray();
                        for(int i = 0; i < 25; i++) {
                            ser.Serialize(writer, new DataItem { Item = i });
                            writer.Flush();
                            Thread.Sleep(500);
                        }
                        writer.WriteEnd();
                        writer.Flush();
                    }
                });
                var readTask = Task.Run(() => {
                    var sw = new Stopwatch();
                    sw.Start();
                    using(var readStream = new AnonymousPipeClientStream(pipeHandle))
                    using(var sr = new StreamReader(readStream))
                    using(var reader = new JsonTextReader(sr)) {
                        var ser = new JsonSerializer();
                        if(!reader.Read() || reader.TokenType != JsonToken.StartArray) {
                            throw new Exception("Expected start of array");
                        }
                        while(reader.Read()) {
                            if(reader.TokenType == JsonToken.EndArray) break;
                            var item = ser.Deserialize<DataItem>(reader);
                            Console.WriteLine("[{0}] Received item: {1}", sw.Elapsed, item);
                        }
                    }
                });
                Task.WaitAll(writeTask, readTask);
                writeStream.DisposeLocalCopyOfClientHandle();
            }
        }

        class DataItem {
            public int Item { get; set; }
            public override string ToString() {
                return string.Format("{{ Item = {0} }}", Item);
            }
        }
    }
}

Note that you may receive an exception when the AnonymousPipeServerStream is disposed, I ignored this as it isn't relevant to the problem at hand.

Rigamarole

I need to modify this so that I can get any complete JSON object. My server and client communicate by sending snippets of JSON so the client could send {"sign in":{"username":"nick"}}{"buy item":{"_id":"32321123"}} and it needs to see this as two fragments of JSON signaling an event each time it reads a fragment. In nodejs this can be done in 3 lines of code. – Hormone

Answer 6

Another option if you're reading in Json is to use the DeserializeObject on the JsonConvert class:

using (StreamReader streamReader = new StreamReader("example.json"))
            {
                string json = streamReader.ReadToEnd();
                ObjectType object = JsonConvert.DeserializeObject<ObjectType>(json);
            }

Lashio

The question was about "take a stream rather than a string", this approach at least doubles the memory consumption (because of the string allocation) compared to direct stream reading. Btw streamReader may be disposed before calling DeserializeObject. – Nat

Answer 7

another option that is handy when you are running out of memory is to periodically flush

/// <summary>serialize the value in the stream.</summary>
/// <typeparam name="T">the type to serialize</typeparam>
/// <param name="stream">The stream.</param>
/// <param name="value">The value.</param>
/// <param name="settings">The json settings to use.</param>
/// <param name="bufferSize"></param>
/// <param name="leaveOpen"></param>
public static void JsonSerialize<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false)
{
    using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize,leaveOpen))
    using (var jsonWriter = new JsonTextWriter(writer))
    {
        var ser = JsonSerializer.Create( settings );
        ser.Serialize(jsonWriter, value);
        jsonWriter.Flush();
    }
}

/// <summary>serialize the value in the stream asynchronously.</summary>
/// <typeparam name="T"></typeparam>
/// <param name="stream">The stream.</param>
/// <param name="value">The value.</param>
/// <param name="settings">The settings.</param>
/// <param name="bufferSize">The buffer size, in bytes, set -1 to not flush till done</param>
/// <param name="leaveOpen"> true to leave the stream open </param>
/// <param name="cancellationToken">Propagates notification that operations should be canceled.</param>
public static Task JsonSerializeAsync<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false, CancellationToken cancellationToken=default)
{
    using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize: bufferSize,leaveOpen: leaveOpen))
    using (var jsonWriter = new JsonTextWriter(writer))
    {
        var ser = JsonSerializer.Create( settings );
        ser.Serialize(jsonWriter, value);
        return  jsonWriter.Flush();
    }
    //jsonWriter.FlushAsnc with my version gives an error on the stream
    return Task.CompletedTask;
}

You can test/ use it like so:

[TestMethod()]
public void WriteFileIntoJsonTest()
{
    var file = new FileInfo(Path.GetTempFileName());
    try
    {
        var list = new HashSet<Guid>();
        for (int i = 0; i < 100; i++)
        {
            list.Add(Guid.NewGuid());
        }
        file.JsonSerialize(list);


        var sr = file.IsValidJson<List<Guid>>(out var result);
        Assert.IsTrue(sr);
        Assert.AreEqual<int>(list.Count, result.Count);
        foreach (var item in result)
        {
            Assert.IsFalse(list.Add(item), $"The GUID {item} should already exist in the hash set");
        }
    }
    finally
    {
        file.Refresh();
        file.Delete();
    }
}

you'd need to create the extension methods, here is the whole set:

public static class JsonStreamReaderExt
    {
       static JsonSerializerSettings _settings ;
        static JsonStreamReaderExt()
        {
            _settings = JsonConvert.DefaultSettings?.Invoke() ?? new JsonSerializerSettings();
            _settings.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
            _settings.DateTimeZoneHandling = DateTimeZoneHandling.Utc;
            _settings.DateFormatHandling = DateFormatHandling.IsoDateFormat ;
        }

    /// <summary>
    /// serialize the value in the stream.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="value">The value.</param>
    public static void JsonSerialize<T>(this Stream stream,[DisallowNull] T value)
    {
        stream.JsonSerialize(value,_settings);
    }

    /// <summary>
    /// serialize the value in the file .
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="file">The file.</param>
    /// <param name="value">The value.</param>
    public static void JsonSerialize<T>(this FileInfo file,[DisallowNull] T value)
    {
        if (string.IsNullOrEmpty(file.DirectoryName)==true && Directory.Exists(file.DirectoryName) == false)
        { 
            Directory.CreateDirectory(file.FullName);
        }
        
        using var s = file.OpenWrite();
        s.JsonSerialize(value, _settings);

        file.Refresh();
    }

    /// <summary>
    /// serialize the value in the file .
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="file">The file.</param>
    /// <param name="value">The value.</param>
    /// <param name="settings">the json settings to use</param>
    public static void JsonSerialize<T>(this FileInfo file, [DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings)
    {
        if (string.IsNullOrEmpty(file.DirectoryName) == true && Directory.Exists(file.DirectoryName) == false)
        {
            Directory.CreateDirectory(file.FullName);
        }

        using var s = file.OpenWrite();
        s.JsonSerialize(value, settings);

        file.Refresh();
    }

    /// <summary>
    /// serialize the value in the file .
    /// </summary>
    /// <remarks>File will be refreshed to contain the new meta data</remarks>
    /// <typeparam name="T">the type to serialize</typeparam>
    /// <param name="file">The file.</param>
    /// <param name="value">The value.</param>        
    /// <param name="cancellationToken">Propagates notification that operations should be canceled.</param>
    public static async Task JsonSerializeAsync<T>(this FileInfo file, [DisallowNull] T value, CancellationToken cancellationToken = default)
    {
        if (string.IsNullOrEmpty(file.DirectoryName) == true && Directory.Exists(file.DirectoryName) == false)
        {
            Directory.CreateDirectory(file.FullName);
        }

        using (var stream = file.OpenWrite())
        {
            await stream.JsonSerializeAsync(value, _settings,bufferSize:1024,leaveOpen:false, cancellationToken).ConfigureAwait(false);
        }
        file.Refresh();
    }
    /// <summary>
    /// serialize the value in the file .
    /// </summary>
    /// <remarks>File will be refreshed to contain the new meta data</remarks>
    /// <typeparam name="T">the type to serialize</typeparam>
    /// <param name="file">The file to create or overwrite.</param>
    /// <param name="value">The value.</param>
    /// <param name="settings">the json settings to use</param>
    /// <param name="cancellationToken">Propagates notification that operations should be canceled.</param>
    public static async Task JsonSerializeAsync<T>(this FileInfo file, [DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, CancellationToken cancellationToken=default)
    {
        if (string.IsNullOrEmpty(file.DirectoryName) == true && Directory.Exists(file.DirectoryName) == false)
        {
            Directory.CreateDirectory(file.FullName);
        }

        using (var stream = file.OpenWrite())
        {
            await stream.JsonSerializeAsync(value, settings,bufferSize:1024,leaveOpen:false, cancellationToken).ConfigureAwait(false);
        }
        file.Refresh();
    }

    /// <summary>serialize the value in the stream.</summary>
    /// <typeparam name="T">the type to serialize</typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="value">The value.</param>
    /// <param name="settings">The json settings to use.</param>
    /// <param name="bufferSize">The buffer size, in bytes, set -1 to not flush till done</param>
    /// <param name="leaveOpen"> true to leave the stream open </param>
    public static void JsonSerialize<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false)
    {
        using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize,leaveOpen))
        using (var jsonWriter = new JsonTextWriter(writer))
        {
            var ser = JsonSerializer.Create( settings );
            ser.Serialize(jsonWriter, value);
            jsonWriter.Flush();
        }
    }

    /// <summary>serialize the value in the stream asynchronously.</summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="value">The value.</param>
    /// <param name="settings">The settings.</param>
    /// <param name="bufferSize">The buffer size, in bytes, set -1 to not flush till done</param>
    /// <param name="leaveOpen"> true to leave the stream open </param>
    /// <param name="cancellationToken">Propagates notification that operations should be canceled.</param>
    public static Task JsonSerializeAsync<T>(this Stream stream,[DisallowNull] T value, [DisallowNull] JsonSerializerSettings settings, int bufferSize=1024, bool leaveOpen=false, CancellationToken cancellationToken=default)
    {
        using (var writer = new StreamWriter(stream,encoding: System.Text.Encoding.UTF32,bufferSize: bufferSize,leaveOpen: leaveOpen))
        using (var jsonWriter = new JsonTextWriter(writer))
        {
            var ser = JsonSerializer.Create( settings );
            ser.Serialize(jsonWriter, value);
            jsonWriter.Flush();
        }
        return Task.CompletedTask;
       
    }



    /// <summary>
    /// Determines whether [is valid json] [the specified result].
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="result">The result.</param>
    /// <returns><c>true</c> if [is valid json] [the specified result]; otherwise, <c>false</c>.</returns>
    public static bool IsValidJson<T>(this Stream stream, [NotNullWhen(true)] out T? result)
    {
        if (stream is null)
        {
            throw new ArgumentNullException(nameof(stream));
        }

        if (stream.Position != 0)
        {
            stream.Seek(0, SeekOrigin.Begin);
        }
        JsonSerializerSettings settings = (JsonConvert.DefaultSettings?.Invoke()) ?? new JsonSerializerSettings() { DateTimeZoneHandling = DateTimeZoneHandling.Utc, DateFormatHandling = DateFormatHandling.IsoDateFormat };
        settings.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
        using (var reader = new StreamReader(stream))
        using (var jsonReader = new JsonTextReader(reader))
        {
            var ser = JsonSerializer.Create(settings);
            try
            {
                result = ser.Deserialize<T>(jsonReader);
            }
            catch { result = default; }
        }
        return result is not null;
    }
    /// <summary>
    /// Determines whether [is valid json] [the specified settings].
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">The stream.</param>
    /// <param name="settings">The settings.</param>
    /// <param name="result">The result.</param>
    /// <returns><c>true</c> if [is valid json] [the specified settings]; otherwise, <c>false</c>.</returns>
    public static bool IsValidJson<T>(this Stream stream, JsonSerializerSettings settings, [NotNullWhen(true)] out T? result)
    {
        if (stream is null)
        {
            throw new ArgumentNullException(nameof(stream));
        }

        if (settings is null)
        {
            throw new ArgumentNullException(nameof(settings));
        }

        if (stream.Position != 0)
        {
            stream.Seek(0, SeekOrigin.Begin);
        }
        using (var reader = new StreamReader(stream))
        using (var jsonReader = new JsonTextReader(reader))
        {
            var ser = JsonSerializer.Create(settings);
            try
            {
                result = ser.Deserialize<T>(jsonReader);
            }
            catch { result = default; }
        }
        return result is not null;
    }
    /// <summary>
    /// Determines whether file contains valid json using the specified settings and reads it into the output.
    /// </summary>
    /// <typeparam name="T">Type to convert into</typeparam>
    /// <param name="file">The file.</param>
    /// <param name="settings">The settings.</param>
    /// <param name="result">The result.</param>
    /// <returns><c>true</c> if [is valid json] [the specified settings]; otherwise, <c>false</c>.</returns>
    /// <exception cref="System.ArgumentNullException">file</exception>
    /// <exception cref="System.ArgumentNullException"></exception>
    /// <exception cref="System.ArgumentNullException">settings</exception>
    /// <exception cref="System.IO.FileNotFoundException">File could not be accessed</exception>
    public static bool IsValidJson<T>(this FileInfo file, JsonSerializerSettings settings, [NotNullWhen(true)] out T? result)
    {
        if (file is null)
        {
            throw new ArgumentNullException(nameof(file));
        }
        if (File.Exists(file.FullName) == false)
        { 
            throw new FileNotFoundException("File could not be accessed",fileName: file.FullName);
        }

        using var stream = file.OpenRead();

        if (stream is null)
        {
            throw new ArgumentNullException(message:"Could not open the file and access the underlying file stream",paramName: nameof(file));
        }



        if (settings is null)
        {
            throw new ArgumentNullException(nameof(settings));
        }


        using (var reader = new StreamReader(stream))
        using (var jsonReader = new JsonTextReader(reader))
        {
            var ser = JsonSerializer.Create(settings);
            try
            {
                result = ser.Deserialize<T>(jsonReader);
            }
            catch { result = default; }
        }
        return result is not null;
    }
    /// <summary>
    /// Determines whether file contains valid json using the specified settings and reads it into the output.
    /// </summary>
    /// <typeparam name="T">Type to convert into</typeparam>
    /// <param name="file">The file.</param>
    /// <param name="result">The result.</param>
    /// <returns><c>true</c> if [is valid json] [the specified result]; otherwise, <c>false</c>.</returns>
    /// <exception cref="System.ArgumentNullException">file</exception>
    /// <exception cref="System.IO.FileNotFoundException">File could not be accessed</exception>
    public static bool IsValidJson<T>(this FileInfo file, [NotNullWhen(true)] out T? result)
    {
        if (file is null)
        {
            throw new ArgumentNullException(nameof(file));
        }
        if (File.Exists(file.FullName) == false)
        { 
            throw new FileNotFoundException("File could not be accessed",fileName: file.FullName);
        }

        JsonSerializerSettings settings =( JsonConvert.DefaultSettings?.Invoke()) ?? new JsonSerializerSettings() { DateTimeZoneHandling= DateTimeZoneHandling.Utc, DateFormatHandling= DateFormatHandling.IsoDateFormat };
        settings.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
        return file.IsValidJson<T>(settings,out result);


    }

} 

Eme