Using JSON Patch to add values to a dictionary

Question

Overview

I'm trying to write a web service using ASP.NET Core that allows clients to query and modify the state of a microcontroller. This microcontroller contains a number of systems that I model within my application - for instance, a PWM system, an actuator input system, etc.

The components of these systems all have particular properties that can be queried or modified using a JSON patch request. For example, the 4th PWM on the micro can be enabled using an HTTP request carrying {"op":"replace", "path":"/pwms/3/enabled", "value":true}. To support this, I'm using the AspNetCore.JsonPatch library.

My problem is that I'm trying to implement JSON Patch support for a new "CAN database" system that logically should map a definition name to a particular CAN message definition, and I'm not sure how to go about this.

Details

The diagram below models the CAN database system. A CanDatabase instance should logically contain a dictionary of the form IDictionary<string, CanMessageDefinition>.

To support creating new message definitions, my application should allow users to send JSON patch requests like this:

{
    "op": "add",
    "path": "/candb/my_new_definition",
    "value": {
        "template": ["...", "..."],
        "repeatRate": "...",
        "...": "...",
    }
}

Here, my_new_definition would define the definition name, and the object associated with value should be deserialised to a CanMessageDefinition object. This should then be stored as a new key-value pair in the CanDatabase dictionary.

The issue is that path should specify a property path which for statically-typed objects would be...well, static (an exception to this is that it allows for referencing array elements e.g. /pwms/3 as above).

What I've tried

A. The Leeroy Jenkins approach

Forget the fact that I know it won't work - I tried the implementation below (which uses static-typing only despite the fact I need to support dynamic JSON Patch paths) just to see what happens.

Implementation

internal sealed class CanDatabaseModel : DeviceComponentModel<CanDatabaseModel>
{
    public CanDatabaseModel()
    {
        this.Definitions = new Dictionary<string, CanMessageDefinition>();
    }

    [JsonProperty(PropertyName = "candb")]
    public IDictionary<string, CanMessageDefinition> Definitions { get; }

    ...
}

Test

{
    "op": "add",
    "path": "/candb/foo",
    "value": {
        "messageId": 171,
        "template": [17, 34],
        "repeatRate": 100,
        "canPort": 0
    }
}

Outcome

An InvalidCastException is thrown at the site where I try to apply the specified changes to the JsonPatchDocument.

Site:

var currentModelSnapshot = this.currentModelFilter(this.currentModel.Copy());
var snapshotWithChangesApplied = currentModelSnapshot.Copy();
diffDocument.ApplyTo(snapshotWithChangesApplied);

Exception:

Unable to cast object of type 'Newtonsoft.Json.Serialization.JsonDictionaryContract' to type 'Newtonsoft.Json.Serialization.JsonObjectContract'.

---

B. Relying on dynamic JSON Patching

A more promising plan of attack seemed to be relying on dynamic JSON patching, which involves performing patch operations on instances of ExpandoObject. This allows you to use JSON patch documents to add, remove or replace properties since you're dealing with a dynamically-typed object.

Implementation

internal sealed class CanDatabaseModel : DeviceComponentModel<CanDatabaseModel>
{
    public CanDatabaseModel()
    {
        this.Definitions = new ExpandoObject();
    }

    [JsonProperty(PropertyName = "candb")]
    public IDictionary<string, object> Definitions { get; }

    ...
}

Test

{
    "op": "add",
    "path": "/candb/foo",
    "value": {
        "messageId": 171,
        "template": [17, 34],
        "repeatRate": 100,
        "canPort": 0
    }
}

Outcome

Making this change allows this part of my test to run without exceptions being raised, but JSON Patch has no knowledge of what to deserialise value as, resulting in the data being stored in the dictionary as a JObject rather than a CanMessageDefinition:

Would it be possible to 'tell' JSON Patch how to deserialise the information by any chance? Perhaps something along the lines of using a JsonConverter attribute on Definitions?

[JsonProperty(PropertyName = "candb")]
[JsonConverter(...)]
public IDictionary<string, object> Definitions { get; }

---

Summary

• I need to support JSON patch requests that add values to a dictionary
• I've tried going down the purely-static route, which failed
• I've tried using dynamic JSON patching
  • This partly worked, but my data was stored as a JObject type instead of the intended type
  • Is there an attribute (or some other technique) I can apply to my property to let it deserialise to the correct type (not an anonymous type)?

Janettjanetta

Implementing a custom JSON deserializer looks like a viable solution. Could you give more details on template in value object? Can we move messageId and template to parent object? – Bumbailiff

@Bumbailiff template represents a CAN message payload (0-8 bytes), so it would be an array of integers. messageId and template have to remain as they are because requests need to adhere to the JSON Patch schema as described in RFC 6902 – Janettjanetta

Did you figure out an approach? This is an interesting scenario and I have bookmarked this to work on when I get some time from work. – Bumbailiff

@Bumbailiff Not yet. I'm using a temporary workaround (registering a PropertyChanged event handler to the ExpandoObject to manually convert the new JObject into a CanMessageDefinition). – Janettjanetta

Leeeeroooooooy! :) – Gourmont

I don't think you can deserialize IDictionary<T, K>. Did you try just using Dictionary<T, K>? I'm using that and not having any issues. – Baseman

Answer 1

Since there doesn't seem to be any official way to do it, I've come up with a Temporary Solution™ (read: a solution that works well enough so I'll probably keep it forever).

In order to make it seem like JSON Patch handles dictionary-like operations, I created a class called DynamicDeserialisationStore which inherits from DynamicObject and makes use of JSON Patch's support for dynamic objects.

More specifically, this class overrides methods like TrySetMember, TrySetIndex, TryGetMember, etc. to essentially act like a dictionary, except that it delegates all these operations to callbacks provided to its constructor.

Implementation

The code below provides the implementation of DynamicDeserialisationStore. It implements IDictionary<string, object> (which is the signature JSON Patch requires to work with dynamic objects) but I only implement the bare minimum of the methods I require.

The problem with JSON Patch's support for dynamic objects is that it will set properties to JObject instances i.e. it won't automatically perform deserialisation like it would when setting static properties, as it can't infer the type. DynamicDeserialisationStore is parameterised on the type of object that it will try to automatically try to deserialise these JObject instances to when they're set.

The class accepts callbacks to handle basic dictionary operations instead of maintaining an internal dictionary itself, because in my "real" system model code I don't actually use a dictionary (for various reasons) - I just make it appear that way to clients.

internal sealed class DynamicDeserialisationStore<T> : DynamicObject, IDictionary<string, object> where T : class
{
    private readonly Action<string, T> storeValue;
    private readonly Func<string, bool> removeValue;
    private readonly Func<string, T> retrieveValue;
    private readonly Func<IEnumerable<string>> retrieveKeys;

    public DynamicDeserialisationStore(
        Action<string, T> storeValue,
        Func<string, bool> removeValue,
        Func<string, T> retrieveValue,
        Func<IEnumerable<string>> retrieveKeys)
    {
        this.storeValue = storeValue;
        this.removeValue = removeValue;
        this.retrieveValue = retrieveValue;
        this.retrieveKeys = retrieveKeys;
    }

    public int Count
    {
        get
        {
            return this.retrieveKeys().Count();
        }
    }

    private IReadOnlyDictionary<string, T> AsDict
    {
        get
        {
            return (from key in this.retrieveKeys()
                    let value = this.retrieveValue(key)
                    select new { key, value })
                    .ToDictionary(it => it.key, it => it.value);
        }
    }

    public override bool TrySetIndex(SetIndexBinder binder, object[] indexes, object value)
    {
        if (indexes.Length == 1 && indexes[0] is string && value is JObject)
        {
            return this.TryUpdateValue(indexes[0] as string, value);
        }

        return base.TrySetIndex(binder, indexes, value);
    }

    public override bool TryGetIndex(GetIndexBinder binder, object[] indexes, out object result)
    {
        if (indexes.Length == 1 && indexes[0] is string)
        {
            try
            {
                result = this.retrieveValue(indexes[0] as string);
                return true;
            }
            catch (KeyNotFoundException)
            {
                // Pass through.
            }
        }

        return base.TryGetIndex(binder, indexes, out result);
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        return this.TryUpdateValue(binder.Name, value);
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        try
        {
            result = this.retrieveValue(binder.Name);
            return true;
        }
        catch (KeyNotFoundException)
        {
            return base.TryGetMember(binder, out result);
        }
    }

    private bool TryUpdateValue(string name, object value)
    {
        JObject jObject = value as JObject;
        T tObject = value as T;

        if (jObject != null)
        {
            this.storeValue(name, jObject.ToObject<T>());
            return true;
        }
        else if (tObject != null)
        {
            this.storeValue(name, tObject);
            return true;
        }

        return false;
    }

    object IDictionary<string, object>.this[string key]
    {
        get
        {
            return this.retrieveValue(key);
        }

        set
        {
            this.TryUpdateValue(key, value);
        }
    }

    public IEnumerator<KeyValuePair<string, object>> GetEnumerator()
    {
        return this.AsDict.ToDictionary(it => it.Key, it => it.Value as object).GetEnumerator();
    }

    public void Add(string key, object value)
    {
        this.TryUpdateValue(key, value);
    }

    public bool Remove(string key)
    {
        return this.removeValue(key);
    }

    #region Unused methods
    bool ICollection<KeyValuePair<string, object>>.IsReadOnly
    {
        get
        {
            throw new NotImplementedException();
        }
    }

    ICollection<string> IDictionary<string, object>.Keys
    {
        get
        {
            throw new NotImplementedException();
        }
    }

    ICollection<object> IDictionary<string, object>.Values
    {
        get
        {
            throw new NotImplementedException();
        }
    }

    void ICollection<KeyValuePair<string, object>>.Add(KeyValuePair<string, object> item)
    {
        throw new NotImplementedException();
    }

    void ICollection<KeyValuePair<string, object>>.Clear()
    {
        throw new NotImplementedException();
    }

    bool ICollection<KeyValuePair<string, object>>.Contains(KeyValuePair<string, object> item)
    {
        throw new NotImplementedException();
    }

    bool IDictionary<string, object>.ContainsKey(string key)
    {
        throw new NotImplementedException();
    }

    void ICollection<KeyValuePair<string, object>>.CopyTo(KeyValuePair<string, object>[] array, int arrayIndex)
    {
        throw new NotImplementedException();
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        throw new NotImplementedException();
    }

    bool ICollection<KeyValuePair<string, object>>.Remove(KeyValuePair<string, object> item)
    {
        throw new NotImplementedException();
    }

    bool IDictionary<string, object>.TryGetValue(string key, out object value)
    {
        throw new NotImplementedException();
    }
    #endregion
}

Tests

The tests for this class are provided below. I create a mock system model (see image) and perform various JSON Patch operations on it.

Here's the code:

public class DynamicDeserialisationStoreTests
{
    private readonly FooSystemModel fooSystem;

    public DynamicDeserialisationStoreTests()
    {
        this.fooSystem = new FooSystemModel();
    }

    [Fact]
    public void Store_Should_Handle_Adding_Keyed_Model()
    {
        // GIVEN the foo system currently contains no foos.
        this.fooSystem.Foos.ShouldBeEmpty();

        // GIVEN a patch document to store a foo called "test".
        var request = "{\"op\":\"add\",\"path\":\"/foos/test\",\"value\":{\"number\":3,\"bazzed\":true}}";
        var operation = JsonConvert.DeserializeObject<Operation<FooSystemModel>>(request);
        var patchDocument = new JsonPatchDocument<FooSystemModel>(
            new[] { operation }.ToList(),
            new CamelCasePropertyNamesContractResolver());

        // WHEN we apply this patch document to the foo system model.
        patchDocument.ApplyTo(this.fooSystem);

        // THEN the system model should now contain a new foo called "test" with the expected properties.
        this.fooSystem.Foos.ShouldHaveSingleItem();
        FooModel foo = this.fooSystem.Foos["test"] as FooModel;
        foo.Number.ShouldBe(3);
        foo.IsBazzed.ShouldBeTrue();
    }

    [Fact]
    public void Store_Should_Handle_Removing_Keyed_Model()
    {
        // GIVEN the foo system currently contains a foo.
        var testFoo = new FooModel { Number = 3, IsBazzed = true };
        this.fooSystem.Foos["test"] = testFoo;

        // GIVEN a patch document to remove a foo called "test".
        var request = "{\"op\":\"remove\",\"path\":\"/foos/test\"}";
        var operation = JsonConvert.DeserializeObject<Operation<FooSystemModel>>(request);
        var patchDocument = new JsonPatchDocument<FooSystemModel>(
            new[] { operation }.ToList(),
            new CamelCasePropertyNamesContractResolver());

        // WHEN we apply this patch document to the foo system model.
        patchDocument.ApplyTo(this.fooSystem);

        // THEN the system model should be empty.
        this.fooSystem.Foos.ShouldBeEmpty();
    }

    [Fact]
    public void Store_Should_Handle_Modifying_Keyed_Model()
    {
        // GIVEN the foo system currently contains a foo.
        var originalFoo = new FooModel { Number = 3, IsBazzed = true };
        this.fooSystem.Foos["test"] = originalFoo;

        // GIVEN a patch document to modify a foo called "test".
        var request = "{\"op\":\"replace\",\"path\":\"/foos/test\", \"value\":{\"number\":6,\"bazzed\":false}}";
        var operation = JsonConvert.DeserializeObject<Operation<FooSystemModel>>(request);
        var patchDocument = new JsonPatchDocument<FooSystemModel>(
            new[] { operation }.ToList(),
            new CamelCasePropertyNamesContractResolver());

        // WHEN we apply this patch document to the foo system model.
        patchDocument.ApplyTo(this.fooSystem);

        // THEN the system model should contain a modified "test" foo.
        this.fooSystem.Foos.ShouldHaveSingleItem();
        FooModel foo = this.fooSystem.Foos["test"] as FooModel;
        foo.Number.ShouldBe(6);
        foo.IsBazzed.ShouldBeFalse();
    }

    #region Mock Models
    private class FooModel
    {
        [JsonProperty(PropertyName = "number")]
        public int Number { get; set; }

        [JsonProperty(PropertyName = "bazzed")]
        public bool IsBazzed { get; set; }
    }

    private class FooSystemModel
    {
        private readonly IDictionary<string, FooModel> foos;

        public FooSystemModel()
        {
            this.foos = new Dictionary<string, FooModel>();
            this.Foos = new DynamicDeserialisationStore<FooModel>(
                storeValue: (name, foo) => this.foos[name] = foo,
                removeValue: name => this.foos.Remove(name),
                retrieveValue: name => this.foos[name],
                retrieveKeys: () => this.foos.Keys);
        }

        [JsonProperty(PropertyName = "foos")]
        public IDictionary<string, object> Foos { get; }
    }
    #endregion
}

Janettjanetta

Answer 2

You could, for instance, deserialize your received Json into an object:

var dataDict = JsonConvert.DeserializeObject<Dictionary<string, object>>(json);

And iterate over it, casting and converting the values of the KeyValuePairs you want to patch into your destination type, CanMessageDefinition:

Dictionary<string, CanMessageDefinition> updateData = new Dictionary<string, CanMessageDefinition>();
foreach (var record in dataDict)
{
    CanMessageDefinition recordValue = (CanMessageDefinition)record.Value;
    if (yourExistingRecord.KeyAttributes.Keys.Contains(record.Key) && (!yourExistingRecord.KeyAttributes.Values.Equals(record.Value)))
    { 
        updateData.Add(record.Key, recordValue);
    }
    
}

And just save your object to your db.

An alternative would be to do this inside a JsonConverter as you mentioned. Cheers

Viviyan