Null-coalescing operator returning null for properties of dynamic objects

Question

I have recently found a problem with the null-coalescing operator while using Json.NET to parse JSON as dynamic objects. Suppose this is my dynamic object:

string json = "{ \"phones\": { \"personal\": null }, \"birthday\": null }";
dynamic d = JsonConvert.DeserializeObject(json);

If I try to use the ?? operator on one of the field of d, it returns null:

string s = "";
s += (d.phones.personal ?? "default");
Console.WriteLine(s + " " + s.Length); //outputs  0

However, if I assign a the dynamic property to a string, then it works fine:

string ss = d.phones.personal;
string s = "";
s += (ss ?? "default");
Console.WriteLine(s + " " + s.Length); //outputs default 7

Finally, when I output Console.WriteLine(d.phones.personal == null) it outputs True.

I have made an extensive test of these issues on Pastebin.

Dm

Can you simplify this a bit? Remove the nested objects, the length and the +=. I doubt they are required to reproduce the issue. – Danyluk

I tried looking at IL, but looking at IL around dynamic is never a pleasure :) – Stpierre

Answer 1

This is due to obscure behaviors of Json.NET and the ?? operator.

Firstly, when you deserialize JSON to a dynamic object, what is actually returned is a subclass of the Linq-to-JSON type JToken (e.g. JObject or JValue) which has a custom implementation of IDynamicMetaObjectProvider. I.e.

dynamic d1 = JsonConvert.DeserializeObject(json);
var d2 = JsonConvert.DeserializeObject<JObject>(json);

Are actually returning the same thing. So, for your JSON string, if I do

    var s1 = JsonConvert.DeserializeObject<JObject>(json)["phones"]["personal"];
    var s2 = JsonConvert.DeserializeObject<dynamic>(json).phones.personal;

Both these expressions evaluate to exactly the same returned dynamic object. But what object is returned? That gets us to the second obscure behavior of Json.NET: rather than representing null values with null pointers, it represents then with a special JValue with JValue.Type equal to JTokenType.Null. Thus if I do:

    WriteTypeAndValue(s1, "s1");
    WriteTypeAndValue(s2, "s2");

The console output is:

"s1":  Newtonsoft.Json.Linq.JValue: ""
"s2":  Newtonsoft.Json.Linq.JValue: ""

I.e. these objects are not null, they are allocated POCOs, and their ToString() returns an empty string.

But, what happens when we assign that dynamic type to a string?

    string tmp;
    WriteTypeAndValue(tmp = s2, "tmp = s2");

Prints:

"tmp = s2":  System.String: null value

Why the difference? It is because the DynamicMetaObject returned by JValue to resolve the conversion of the dynamic type to string eventually calls ConvertUtils.Convert(value, CultureInfo.InvariantCulture, binder.Type) which eventually returns null for a JTokenType.Null value, which is the same logic performed by the explicit cast to string avoiding all uses of dynamic:

    WriteTypeAndValue((string)JsonConvert.DeserializeObject<JObject>(json)["phones"]["personal"], "Linq-to-JSON with cast");
    // Prints "Linq-to-JSON with cast":  System.String: null value

    WriteTypeAndValue(JsonConvert.DeserializeObject<JObject>(json)["phones"]["personal"], "Linq-to-JSON without cast");     
    // Prints "Linq-to-JSON without cast":  Newtonsoft.Json.Linq.JValue: ""

Now, to the actual question. As husterk noted the ?? operator returns dynamic when one of the two operands is dynamic, so d.phones.personal ?? "default" does not attempt to perform a type conversion, thus the return is a JValue:

    dynamic d = JsonConvert.DeserializeObject<dynamic>(json);
    WriteTypeAndValue((d.phones.personal ?? "default"), "d.phones.personal ?? \"default\"");
    // Prints "(d.phones.personal ?? "default")":  Newtonsoft.Json.Linq.JValue: ""

But if we invoke Json.NET's type conversion to string by assigning the dynamic return to a string, then the converter will kick in and return an actual null pointer after the coalescing operator has done its work and returned a non-null JValue:

    string tmp;
    WriteTypeAndValue(tmp = (d.phones.personal ?? "default"), "tmp = (d.phones.personal ?? \"default\")");
    // Prints "tmp = (d.phones.personal ?? "default")":  System.String: null value

This explains the difference you are seeing.

To avoid this behavior, force the conversion from dynamic to string before the coalescing operator is applied:

s += ((string)d.phones.personal ?? "default");

Finally, the helper method to write the type and value to the console:

public static void WriteTypeAndValue<T>(T value, string prefix = null)
{
    prefix = string.IsNullOrEmpty(prefix) ? null : "\""+prefix+"\": ";

    Type type;
    try
    {
        type = value.GetType();
    }
    catch (NullReferenceException)
    {
        Console.WriteLine(string.Format("{0} {1}: null value", prefix, typeof(T).FullName));
        return;
    }
    Console.WriteLine(string.Format("{0} {1}: \"{2}\"", prefix, type.FullName, value));
}

(As an aside, the existence of the null-type JValue explains how the expression (object)(JValue)(string)null == (object)(JValue)null might possibly evaluate to false).

Laicize

By the way ordinary Conditional operator works just fine as well: d.phones.personal != null ? d.phones.personal : "default". I assume that not equals does cast to string first and that's why it works – Aloeswood

A funny comment: Visual Studio does not know about this and suggests to switch Conditional operator to Null Coalescence directly leading you to a bug. Resharper on the other hands knows that and suggests it only when the arguments are not dynamic :) One can not outsmart R# – Aloeswood

What a great answer! Congratulations! – Marabou

Answer 2

I think that I figured out the reason for this... It looks as though the null coalescing operator converts the dynamic property into a type that matches the output type of the statement (in your case it performs a ToString operation on the value of d.phones.personal). The ToString operation converts the "null" JSON value to be an empty string (and not an actual null value). Thus, the null coalescing operator sees the value in question as an empty string instead of null which causes the test to fail and the "default" value is not returned.

More info: https://social.msdn.microsoft.com/Forums/en-US/94b3ca1c-bbfa-4308-89fa-6b455add9de6/dynamic-improvements-on-c-nullcoalescing-operator?forum=vs2010ctpvbcs

Also, when you inspect the dynamic object with the debugger you can see that it shows the value for d.phones.personal as "Empty" and not null (see image below).

A possible workaround for this issue is to safely cast the object prior to performing the null coalescing operation as in the sample below. This will prevent the null coalescing operator from performing the implicit casting.

string s = (d.phones.personal as string) ?? "default";

Hiddenite

Answer 3

While all the above answers are correct, the workaround can be cumbersome if you have large JSON structures with many members. This is due to the way JSON.NET is designed.

One solution I found was to use an alternative to JSON.NET which natively supports null coalescing operators and the library can be easily replaced with minimal code changes as described here:

https://mcmap.net/q/823319/-how-to-use-null-conditional-operator-in-a-dynamic-json-with-jsonconvert-deserializeobject

Retractile