How to deserialize stream to object using System.Text.Json APIs

Question

I'm receiving a response from a web api call as a stream and need to deserialize it to a model.

This is a generic method, so I can't say which parts of code will use this and what's the response payload.

Here's the method:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var serializer = new JsonSerializer();
    using var streamReader = new StreamReader(response);
    using var reader = new JsonTextReader(streamReader);
    return serializer.Deserialize<T>(reader);
}

I'm trying to remove Newtonsoft and use System.Text.Json API.

I found this porting guide in corefx repo in Github, where section Reading from a Stream/String states:

We currently (as of .NET Core 3.0 preview 2) do not have a convenient API to read JSON from a stream directly (either synchronously or asynchronously). For synchronous reading (especially of small payloads), you could read the JSON payload till the end of the stream into a byte array and pass that into the reader

So following this advise I come up with the following:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var length = response.Length;
    var buffer = ArrayPool<byte>.Shared.Rent((int)length);
    var memory = new Memory<byte>(buffer);
    await response.WriteAsync(memory);
    var result = JsonSerializer.Deserialize<T>(memory.Span);
    ArrayPool<byte>.Shared.Return(buffer);
    return result;
}

So my question is - did I understand the advise correctly and this is the way to go ?

This implementation probably can be improved on many aspects, but what bothers me most is renting the byte array from the pool e.g. Stream.Length is a long and I convert it to int which can cause OverflowException.

I tried to look into System.IO.Pipelines and use ReadOnlySequence<byte> overloads of JSON API, but it gets very complicated.

Duffy

Answer 1

I believe that documentation needs to be updated because .NET Core 3 has a method to read from a stream directly. Using it is straight-forward, assuming the stream is encoded in UTF8:

private static readonly JsonSerializerOptions Options = new JsonSerializerOptions();

private static async Task<T> Deserialize<T>(HttpResponseMessage response)
{
    var contentStream = await response.Content.ReadAsStreamAsync();
    var result = await JsonSerializer.DeserializeAsync<T>(contentStream, Options);
    return result;
}

One thing to watch out for is that by default HttpClient will buffer the response content in-memory before returning unless you set the HttpCompletionOption to ResponseHeadersRead when invoking SendAsync:

var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);

Nuriel

Thank you for response. I saw later this API, though now I'm questioning what about bigger payloads. Chopping up entire stream and reading in buffers is done by JsonTextReader. Doesn't seem to be the case with System.Text.Json API, but that's another question. – Duffy

Hi @Mike, did you find a solution for the bigger payloads? – Shantell

@DirkBoer not sure which .NET Core version you're using, but in our case we're on .NET Core 3.1 & the implementation of DeserializeAsync from stream takes care of chopping up the stream to buffers and reading from them. – Duffy

Answer 2

Starting from ASP.NET Core 6 you should use DeserializeAsyncEnumerable() method:

using var request = new HttpRequestMessage(HttpMethod.Get, "/api/products");
request.SetBrowserResponseStreamingEnabled(true); // Enable response streaming
using var response = await Http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
IAsyncEnumerable<Product?> products = JsonSerializer.DeserializeAsyncEnumerable<Product>(stream, new JsonSerializerOptions
    {
       PropertyNameCaseInsensitive = true,
       DefaultBufferSize = 128
     });

 await foreach (Product? product in products)
 {
   // ...
 }
}

Escolar

Great point re DeserializeAsEnumerable. The only minor thing is the request.SetBrowserResponseStreamingEnabled method is part of the webAssembly/Blazor project, and the question relates to a Web API project (so sending request from the server) – Fiann

I get an error in my deserialized object with this method! (Id = 12, Status = Faulted, Method = "{null}", Result = "{Not yet computed}") – Equate