MissingMethodException with Newtonsoft.Json when using TypeNameAssemblyFormat with PCLs

Question

Is there a problem with using the TypeNameAssemblyFormat with PCLs? I have no problems using any other settings with Newtonsoft.Json except when I use this serialization setting.

Here is my Json-related code:

var settings = new JsonSerializerSettings()
        {
            TypeNameHandling = TypeNameHandling.Objects,
            Formatting = Formatting.Indented,
            TypeNameAssemblyFormat = System.Runtime.Serialization.Formatters.FormatterAssemblyStyle.Full
        };

var json = JsonConvert.SerializeObject(obj, settings);

var jsonBytes = Encoding.UTF8.GetBytes(json);

return jsonBytes;

When I make the call within the same library where it is declared, it is fine. However, when I make the call from a different PCL that calls the above code, I get the missing method exception. This only occurs when I use the TypeNameAssemblyFormat setting (i.e. if I didn't have to use that setting then I wouldn't be writing this post ;).

I am using PCL profile 7.

Exception (I didn't want to blah the entire stack trace, but I can if anyone thinks that would help):

"System.MissingMethodException: Method not found: 'Void Newtonsoft.Json.JsonSerializerSettings.set_TypeNameAssemblyFormat(System.Runtime.Serialization.Formatters.FormatterAssemblyStyle)'

Pilchard

In your test, when you get above exception, is Newtonsoft.Json referenced only from first the PCL where it used, or are there other non-PCL assemblies (or may be even PCL assemblies with different profile) which also reference Newtonsoft.Json. In other words, when this code doesn't work, which all assemblies reference Newtonsoft.Json, and what are their profiles? On the flip side, when it does work, is Newtonsoft.Json referenced only from one assembly, the PCL which contains above code? – Entree

Newtonsoft is referenced from PCL1, PCL2, Test1 and Test2(.net 4.5). Passes on Test1 that references PCL1, fails on Test2 that references PCL1, PCL2. I don't think the project would compile if each of these did not reference Newtonsoft. – Pilchard

Scratch the above comment: Newtonsoft is referenced from PCL1, PCL2, Test1 and Test2(.net 4.5). Fails on both Test1 and Test2. I did several days of troubleshooting this and I don't remember at what point I asked this SO question, and I don't remember the specific circumstances of success/failure. After retesting some this morning, both Test1 and Test2 fail. – Pilchard

For what it's worth: I was getting the same missing method exception in a Windows Phone 8.1 project (no explicit use of JsonSerializerSettings) when using Newtonsoft 8.0.3, ostensibly compatible for PCL. Newtonsoft was referencing mscorlib which isn't available to either PCL or Windows Phone. I rolled back my nuget reference to 8.0.2, rebuilt, and the problem went away. – Prelect

Answer 1

Although there isn't enough information in the question to confirm the root cause with 100% confidence.. Personally, after some experimentation I am positive that the only plausible explanation is as follows -

In short - In the test which fails, the correct (portable) version of Newtonsoft.Json.dll is not getting loaded.

In long - There are two tests being performed.

1. Passes - I presume an exe, which calls into PCL1, which calls into portable version of NewtonSoft.Json.dll

2. Fails - I presume another exe, which calls into PCL2, which calls into PCL1, which calls into a version (which version?) of NewtonSoft.Json.dll

The issue is not that PCL2 calls into PCL1 and somehow fails, due to indirect call being made into NewtonSoft.Json.dll. Rather, the issue is, as I am trying to highlight above, the second test happens to be setup in a way, that the wrong / non-portable version of NewtonSoft.Json.dll is available for PCL1 to consume.

In scenario which fails, imagine that the exe or any other non-portable assembly of that application also takes a dependency on (non-portable) version of NewtonSoft.Json.dll. In that case, in the output folder of the application / exe, there will be only one version of NewtonSoft.Json.dll, and if it is the non-portable one, then it will fail with above mentioned exception..

Further explanation - Why?

The type System.Runtime.Serialization.Formatters.FormatterAssemblyStyle is typically defined in mscorlib.dll. However, this type is not available for Portable class libraries to use (don't know about all profiles, but there are some profiles for sure, which do not have this type available). Hence the portable version of NewtonSoft.Json.dll, declares it itself, in it's own assembly.

Check the decompiled version of portable NewtonSoft.Json.dll in your favorite decompiler. Note line number 3 below.. following snippet is from NewtonSoft.Json.dll.

// Decompiled with JetBrains decompiler
// Type: System.Runtime.Serialization.Formatters.FormatterAssemblyStyle
// Assembly: Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
// Assembly location: C:\Users\.....\packages\Newtonsoft.Json.6.0.7\lib\portable-net40+sl5+wp80+win8+wpa81\Newtonsoft.Json.dll
// Compiler-generated code is shown

namespace System.Runtime.Serialization.Formatters
{
  /// <summary>
  /// Indicates the method that will be used during deserialization for locating and loading assemblies.
  /// 
  /// </summary>
  public enum FormatterAssemblyStyle
  {
    Simple,
    Full,
  }
}

Now, when you compile code in a PCL, which references TypeNameAssemblyFormat property, presumeable of type System.Runtime.Serialization.Formatters.FormatterAssemblyStyle defined in Newtonsoft.Json.dll, following IL is generated (decompiled using Ildasm) -

  IL_0017:  ldloc.0
  IL_0018:  ldc.i4.1
  IL_0019:  callvirt   instance void [Newtonsoft.Json]Newtonsoft.Json.JsonSerializerSettings::set_TypeNameAssemblyFormat(valuetype [Newtonsoft.Json]System.Runtime.Serialization.Formatters.FormatterAssemblyStyle)

Note how reference to the type is qualified with the assembly name [Newtonsoft.Json] (scroll to the right -----> to see it on the FormatterAssemblyStyle parameter being passed).

Now, if this portable version of Newtonsoft.Json.dll, gets replaced with non-portable version (because other parts of the project reference non-portable version), then at runtime, CLR will be unable to find method which matches the above signature (as seen in IL above).. and hence fail with System.MissingMethodException.

Unfortunately, the exception itself doesn't give enough information about the complete and exact signature of method it is looking for, including the assembly names.. The type name alone deceptively looks like something that would exist in one of the system dlls (mscorlib.dll in this case).. and not portable version of Newtonsoft.json.dll.

Entree

I gave up on this one awhile back. I will look at it again today and try to give information based on your answer. – Pilchard

From memory, however, I can tell you that in the success case, it is the .Tests unit test project that is executing, and it is targeting .Net 4.5. In one of the failure cases, the .Tests unit test project is also targeting .Net 4.5. The only difference was that the second unit test project references two PCLs, both with the same profile that when loaded with Xamarin Studio show profile 7 (Tho with these projects, I develop in VS. It however doesn't have a profile indicator. In VS I see .Net 4.5, Win 8, Xamarin.Android/iOS/iOS (Classic) checked in project properties in both PCLs). – Pilchard

After retesting, it turns out that my memory has failed me. It fails in both Test1 and Test2 unit test projects. (In my defense, this was asked a month-and-a-half ago!) There is indeed only one version of the Newtonsoft dll in the output directory. After reviewing your answer, I would agree with your assessment of the situation. But the problem still remains. Is there a way of getting it to work? Btw, I'll go ahead and mark your response as the answer as it clearly identifies the situation. For anyone else interested, I'm using ServiceStack.Text now and it has been working so far. – Pilchard

I've completed the answer below. It was too much text/code to include in the comment. – Pilchard

Thanks for the effort put into this answer. I don't think it's exactly what I've been experiencing but it's a good theory and very useful anyway. – Meretricious

+1 I have experienced this happening when you have a solution with multiple projects where there is an inconsistency in the way Newtonsoft.Json.dll is referenced. In my case, all projects were net45 class libraries (plus a single net45 console app) - however, since a subset of the projects referenced the portable version of Newtonsoft.Json.dll (via .csproj and packages.config), the compiler chose to include the portable version, which then gave me this runtime error. After cleaning up the inconsistencies, it worked. Thanks for having done a great job on describing your findings :) – Interplanetary

Answer 2

Ok, I've ended up completing the answer of my own question due to the clarity of Vikas' framing of the problem. And the resolution is the standard PCL approach with this type of problem: Create interface, configure container, use DI.

In this case, in my PCL, I created an INewtonsoftJsonSettingsProvider interface that has the two settings I use as properties as follows:

public interface INewtonsoftJsonSettingsProvider
{
    JsonSerializerSettings Default { get; set; }
    JsonSerializerSettings Concrete { get; set; }
}

Then, in my PCL, I create a concrete implementation of this class as follows:

public class NewtonsoftJsonSettingsProvider : Interfaces.INewtonsoftJsonSettingsProvider
{
    public JsonSerializerSettings Default { get; set; }
    public JsonSerializerSettings Concrete { get; set; }
}

Note: I could easily skip the interface and just use this helper class alone, but I like to use interfaces when dealing with the container.

Then, in my PCL where my Newtonsoft serializer exists, I consume the settings from the container instead of directly creating those inside of the serialization methods. I'll go ahead and include that code here too (I abstracted serialization to an interface because of this problem, so I can swap out implementations):

public class NewtonsoftJsonSerializer : ICustomSerializer
{
    public static void RegisterAsSerializerInContainer()
    {
        var key = Resx.DIKeys.DefaultSerializer;
        var typeContract = typeof(ICustomSerializer);

        if (DI.Ton.KeyExists(key))
        {
            var instance = DI.Ton.Get(typeContract, key);
            DI.Ton.RemoveInstance(key, instance, typeContract);
        }

        DI.Ton.AddImplementationType(typeof(ICustomSerializer), 
                                     typeof(NewtonsoftJsonSerializer), 
                                     isShared: true);

        var serializer = new NewtonsoftJsonSerializer();
        DI.Ton.AddInstance<ICustomSerializer>(Resx.DIKeys.DefaultSerializer, serializer);
    }

    /// <summary>
    /// This is used to overcome the problem of PCL vs non-PCL versions when using TypeNameAssemblyFormat.
    /// see https://mcmap.net/q/1086111/-missingmethodexception-with-newtonsoft-json-when-using-typenameassemblyformat-with-pcls
    /// </summary>
    /// <returns></returns>
    private static INewtonsoftJsonSettingsProvider GetSettingsProvider()
    {
        var key = typeof(INewtonsoftJsonSettingsProvider).Name;
        var settings = DI.Ton.GetInstance<INewtonsoftJsonSettingsProvider>(key);
        return settings;
    }

    public byte[] SerializeToBytes(object obj)
    {
        var settings = GetSettingsProvider();

        var json = JsonConvert.SerializeObject(obj, settings.Default);

        var jsonBytes = Encoding.UTF8.GetBytes(json);

        return jsonBytes;
    }


    public T DeserializeFromBytes<T>(byte[] serializedBytes)
    {
        var json = Encoding.UTF8.GetString(serializedBytes, 0, serializedBytes.Length);

        var settings = GetSettingsProvider();

        var obj = JsonConvert.DeserializeObject<T>(json, settings.Default);

        return obj;
    }

    public byte[] SerializeToBytes_UseConcreteTypes(object obj)
    {
        var settings = GetSettingsProvider();

        var json = JsonConvert.SerializeObject(obj, settings.Concrete);

        var jsonBytes = Encoding.UTF8.GetBytes(json);

        return jsonBytes;
    }

    public T DeserializeFromBytes_UseConcreteTypes<T>(byte[] serializedBytes)
    {
        var json = Encoding.UTF8.GetString(serializedBytes, 0, serializedBytes.Length);

        var settings = GetSettingsProvider();

        var obj = JsonConvert.DeserializeObject<T>(json, settings.Concrete);

        return obj;
    }
}

Then, in my consuming non-PCL and non-Xamarin (may work in PCL but Xamarin has problem - see below), I configure the container with the proper System.Runtime.Serialization.Formatters.FormatterAssemblyStyle as explained in Vikas' answer:

private static void UseNewtonsoft()
{
    var defaultNewtonsoftSettings = new Newtonsoft.Json.JsonSerializerSettings()
    {
        Formatting = Newtonsoft.Json.Formatting.Indented
    };
    var concreteNewtonsoftSettings = new Newtonsoft.Json.JsonSerializerSettings()
    {
        TypeNameHandling = TypeNameHandling.Objects,
        Formatting = Formatting.Indented,
        TypeNameAssemblyFormat = System.Runtime.Serialization.Formatters.FormatterAssemblyStyle.Full
    };
    Serialization.NewtonsoftJsonSettingsProvider.CreateAndRegisterWithContainerIdem(defaultNewtonsoftSettings, concreteNewtonsoftSettings);
    Serialization.NewtonsoftJsonSerializer.RegisterAsSerializerInContainer();
}

This executes without a problem in my .Net test projects. However, when I went to use this in Xamarin.Android project, I got an error stating that the FormatterAssemblyStyle exists in both Newtonsoft and in MonoAndroid mscorlib. Since Xamarin Studio doesn't seem to do extern aliases, I used reflection and dynamic as follows:

void UseNewtonsoft()
{
    var defaultNewtonsoftSettings = new Newtonsoft.Json.JsonSerializerSettings()
        {
            Formatting = Newtonsoft.Json.Formatting.Indented
        };
    //hack: FormatterAssemblyStyle exists in two dlls and extern alias doesn't work in xamarin studio
    var assmNewtonsoft = System.Reflection.Assembly.GetAssembly(typeof(Newtonsoft.Json.ConstructorHandling));
    var enumType = assmNewtonsoft.GetType("System.Runtime.Serialization.Formatters.FormatterAssemblyStyle");
    dynamic enumInstance = Enum.Parse(enumType, "Full");
    var concreteNewtonsoftSettings = new Newtonsoft.Json.JsonSerializerSettings()
        {
            TypeNameHandling = Newtonsoft.Json.TypeNameHandling.Objects,
            Formatting = Newtonsoft.Json.Formatting.Indented,
            //TypeNameAssemblyFormat = System.Runtime.Serialization.Formatters.FormatterAssemblyStyle.Full //set using dynamic
        };
    dynamic dynSettings = concreteNewtonsoftSettings;
    dynSettings.TypeNameAssemblyFormat = enumInstance;
    commonGib.Serialization.NewtonsoftJsonSettingsProvider.CreateAndRegisterWithContainerIdem(defaultNewtonsoftSettings, concreteNewtonsoftSettings);
    commonGib.Serialization.NewtonsoftJsonSerializer.RegisterAsSerializerInContainer();
}

Pilchard

I think the root of the problem is either a bug in Json.NET or NuGet or both. I've awarded you the bounty because it actually solves the problem and I was running out of time to award the bounty to someone. Thanks for the effort you've put in. – Meretricious

@Meretricious I don't think this is bug in Json.Net or Nuget.. May be we can discuss it in chat.. because I think there are variations of the essence of the problem which you will see in other places too.. to me, best analogy is DLL hell.. i.e. version conflicts. I am glad that you awarded the bounty to ibgib because DI does seem to be a very good solution to the actual problem, which I couldn't confirm with 100% confidence. If your problem is indeed slightly different, I hope you can put another question with more information.. and we'll see if we can help. – Entree

@VikasGupta Yes, absolutely I will put together an explanation of what I've discovered myself about the issue. Unfortunately I just haven't got around to it yet. What I have done though, is set myself a reminder to get back to it so I don't forget. – Meretricious

It feels somewhat like a bug or poor design decision, but perhaps it's just a necessary evil...which extends further to another problem as well: My answer works for .Net test projects, but when I go to use this approach in my Xamarin.Android project, it gives me the error that the type is defined in both Newtonsoft and in MonoAndroid's mscorlib. I am still working on fixing this (I don't know how to scope to one in particular). – Pilchard

I have edited my answer to work with Xamarin.Android. – Pilchard