Newtonsoft.Json - Out of memory exception while deserializing big object

Question

I have a problem deserializing a JSON file of about 1GB. When I run the following code I get an out of memory exception:

using (FileStream sr = new FileStream("myFile.json", FileMode.Open, FileAccess.Read))
{
  using (StreamReader reader = new StreamReader(sr))
  {
    using (JsonReader jsReader = new JsonTextReader(reader))
    {
      JsonSerializer serializer = new JsonSerializer();
      dataObject = serializer.Deserialize<T>(jsReader);
    }
  }
}

the exception is thrown by

Newtonsoft.Json.Linq.JTokenWriter.WriteValue(Int64 value)

The serialization works well, here is the code I'm using

using (StreamWriter reader = new StreamWriter("myFile.json"))
{
   using (JsonReader jsWriter = new JsonWriter(reader))
   {
      JsonTextWriter jsonWriter = new JsonTextWriter(jsWriter) { Formatting = Formatting.Indented };
      JsonSerializer ser = new JsonSerializer();
      ser.Serialize(jsonWriter, dataObject, dataObject.GetType());
      jsonWriter.Flush();
    }
}}

Am I doing something wrong in the deserialization? Can you help suggesting a way to deserialize big json object?

Thanks

Sharanshard

1) Make sure you are running as a 64 bit application. If you are running in 32 bit mode you probably just ran out of memory. 2) What is the full traceback of the out of memory exception? Are you running out of memory in a JsonConverter? – Paphlagonia

I'm running a 32 bit application, and I'm running in the out of memory executing the line dataObject = serializer.Deserialize<T>(jsReader); and the function that throws the exception is Newtonsoft.Json.Linq.JTokenWriter.WriteValue(Int64 value) – Sharanshard

Answer 1

According to Newtonsoft.Json Performance Tips your approach has to work (because you read via stream and it should make portion from your file). I can't figure out why your code doesn't work.

But you can try another approach, that was described in the next article - Parsing Big Records with Json.NET

Galatea

Probably a 1GB Json deserialized into an object does not "fit" in memory – Rating

Answer 2

I know this is an old question, but I´ve just stepted over this same problem with a 50GB json file. Here is my solution:

using (FileStream sr = new FileStream(jsonFile, FileMode.Open, FileAccess.Read))
using (StreamReader fileReader = new StreamReader(sr))
using (JsonTextReader jsonReader = new JsonTextReader(fileReader))
{
    JsonSerializer jsonSerializer = new JsonSerializer();
    List<JObject> documentBatch = new List<JObject>();

    while (jsonReader.Read())
    {
        if (jsonReader.TokenType == JsonToken.StartObject)
        {                   
            JObject document = jsonSerializer.Deserialize<JObject>(jsonReader);             
            documentBatch.Add(document);
            
            if (documentBatch.Count >= 1000) // Adjust the batch size as needed
            {
                foreach (JObject item in documentBatch)
                {
                    //do your stuff
                }
                
                documentBatch.Clear();
            }               
        }
    }
}

Ragucci