"Current error context error is different to requested error" exception in Json.Net

Question

Context

I've written a parallel job framework in C# to import/export a large amount of data from/to an ElasticSearch cluster. To do this I've modelled each import or export of a single item as an object that gets executed at some point by the framework. To interface with ElasticSearch I'm using NEST (official .NET ElasticSearch client library) v1.7.1 and JSON.Net 7.0.1.

Each of the import/export task objects interacts with ElasticSearch using NEST. For performance reasons, I have written a proxy class which groups search requests generated by the task objects into fixed-size batches to use with NEST's _msearch API. The caller to this class is delayed until its batch returns. That class is available here.

My framework wraps models the result of each import/export task as either "bool" or "Exception". The overall process is able to continue even if errors with individual items are encountered.

Problem

I am seeing the following exception raised thousands of times after several hours of tasks completing without errors:

System.InvalidOperationException: Current error context error is different to requested error.
    at _____.Matcher.<GetBestMatchAsync>d__15.MoveNext() in C:\\_work\\edc7a363\\_____\\Matcher.cs:line 266
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
    _____.MatchBlock`1.<ExecuteAsyncInternal>d__19.MoveNext() in C:\\_work\\edc7a363\\_____\\MatchBlock.cs:line 111
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
    at _____.Block.BlockBase.<ExecuteAsync>d__11.MoveNext() in C:\\_work\\edc7a363\\_____\\Block\\BlockBase.cs:line 33

This is the code throwing the exception (from the bulk searcher class linked above):

try
{
    var bulkResponse = Client.MultiSearch(searchDescriptor);
    var items = bulkResponse.GetResponses<T>().ToList();

    // Set response values and release all waiting tasks
    var zip = currentBuffer.Zip(items, (op, result) => new { op, result });
    foreach (var a in zip)
    {
        a.op.Response = a.result;
        a.op.Cts.Cancel();
    }
}
catch (Exception e)
{
    foreach (var op in currentBuffer)
    {
        op.Error = e;
        op.Cts.Cancel();
    }
}

where Client is an IElasticClient.

Googling the exception message leads me to this method in the JsonSerializerInternalBase class in JSON.Net, which seems to be executed after each deserialisation:

private ErrorContext GetErrorContext(object currentObject, object member, string path, Exception error)
{
    if (_currentErrorContext == null)
    {
        _currentErrorContext = new ErrorContext(currentObject, member, path, error);
    }

    if (_currentErrorContext.Error != error)
    {
        throw new InvalidOperationException("Current error context error is different to requested error.");
    }

    return _currentErrorContext;
}

Given a single NEST object is being reused for every operation across multiple threads - and I think NEST only uses one JsonSerializer instance - this makes me think this part of JSON.Net is not thread-safe. Though it's strange how the error doesn't start happening until a few hours into a run.

How can I debug this further?

Inspection

JsonSerializerInternalBase is the base class for JsonSerializerInternalWriter.cs and JsonSerializerInternalReader.cs not JsonSerializer. For each call to Serialize, Deserialize or Populate, JsonSerializer allocates one of these to do the actual work -- likely for thread safety. – Tizes

1) That traceback looks incomplete - it doesn't show any of the traceback inside Json.NET itself. Can you edit the question to include the entire traceback including all inner exceptions? 2) Can you show the call to serializer.Serialize() including how JsonSerializer.Error is initialized? – Tizes

1. That stack trace is the one I have. The block of code from the bulk searcher class copies the exception out and is rethrown on the caller thread; I guess it loses the full stack trace. I will change the code to produce the full trace and run it overnight. 2. NEST creates its own JsonSerializer, though it's complicated by my use of a custom JsonConverter that calls onto a different JsonSerializer. There is a lot of code here so I will wait for the results from the re-run before I start formatting it for StackOverflow. Thanks for your Qs – Inspection

Answer 1

My colleague eventually traced the error down - it was due to an exception being thrown from inside a JsonConverter which was called by another JsonConverter. The "error context" is an internal JSON.Net thing used to track the last exception thrown. Seems like the exception was handled by the wrong JsonConverter. We added a flag to the inner JsonConverter to let it know not to throw the exception in a certain context.

Inspection

Can you please provide some code example? – Alverta

Answer 2

I hit this error when an enum property was being serialised as null with a custom serialiser, then deserialised with the default deserialiser.

Fickle

Answer 3

I found this can also happen if you are using the JsonSerializerSettings error handler like this:

parsedResponse = JsonConvert.DeserializeObject<T>(
  json,
  new JsonSerializerSettings
  {
    Error = (object sender, ErrorEventArgs args) =>
    {
      throw new MyCustomException(String.Format("Parse error: {0}", args.ErrorContext.Error.Message));
    },
...

The thrown error will not be MyCustomException, but rather the "Current error context error is different to requested error". Apparently Json.NET doesn't like it if you throw an exception inside the error handler

Stila

Since the ErrorContext.Error is the last serialization exception thrown, any exception thrown inside the JavaSerializationSettings.Error event handler will cause the "Current error context error is different to requested error" error b/c json.net catches any exception that occurs during serialization/deserialization and compares it to the last serialization exception that was thrown. In this case, the exception thrown inside the event handler won't match the last serialization exception. – Eau