System.Text.Json and Dynamically Parsing polymorphic objects

Question

I don't believe I am wrapping my head around how to properly use JsonConverter for polymorphism in parsing json results.

In my scenario, I am targeting Git Policy Configurations in TFS. A policy configuration:

"value": [
{
        "createdBy": {
            "displayName": "username",
            "url": "url",
            "id": "id",
            "uniqueName": "user",
            "imageUrl": "url"
        },
        "createdDate": "2020-03-21T18:17:24.3240783Z",
        "isEnabled": true,
        "isBlocking": true,
        "isDeleted": false,
        "settings": {
            "minimumApproverCount": 1,
            "creatorVoteCounts": false,
            "allowDownvotes": false,
            "resetOnSourcePush": true,
            "scope": [{
                    "refName": "refs/heads/master",
                    "matchKind": "Exact",
                    "repositoryId": "id"
                }
            ]
        },
        "_links": {
            "self": {
                "href": "url"
            },
            "policyType": {
                "href": "url"
            }
        },
        "revision": 1,
        "id": 974,
        "url": "url",
        "type": {
            "id": "id",
            "url": "url",
            "displayName": "Minimum number of reviewers"
        },
{...}]

More settings examples: Require a Merge Strategy

"settings": {
        "useSquashMerge": true,
        "scope": [
            {
                "refName": "refs/heads/master",
                "matchKind": "Exact",
                "repositoryId": "id"
            }
        ]
    }

Required Reviewers

    "settings": {
        "requiredReviewerIds": [
            "id"
        ],
        "scope": [
            {
                "refName": "refs/heads/master",
                "matchKind": "Exact",
                "repositoryId": "id"
            }
        ]
    }

In the json snippet above, the settings object is different based on the type of configuration.

What is the best approach to writing a converter than can dynamically serialize/deserialize the settings object? I've read a couple of articles regarding this and can't quite wrap my head around it.

---

This is how I am currently deserializing all of my API results, so far they have been simple result sets.

async Task<List<T>> ParseResults<T>( HttpResponseMessage result, string parameter )
{
    List<T> results = new List<T>();

    if ( result.IsSuccessStatusCode )
    {
        using var stream = await result.Content.ReadAsStreamAsync();
        JsonDocument doc = JsonDocument.Parse( stream );
        JsonElement collection = doc.RootElement.GetProperty( parameter ).Clone();

        foreach ( var item in collection.EnumerateArray() )
        {
            results.Add( JsonSerializer.Deserialize<T>( item.ToString() ) );
        }
    }

    return results;
}

My integration test.

PolicyConfiguration is the type I am trying to deserialize to.

[Test]
public async Task Get_TestMasterBranchPolicyConfigurations()
{
    HttpResponseMessage result = await GetResult( $"{_collection}/ProductionBuildTesting/_apis/policy/configurations?api-version=4.1" );

    List<PolicyConfiguration> configurations = await ParseResults<PolicyConfiguration>( result, "value" );
    Assert.AreEqual( 16, configurations.Count );
    JsonPrint( configurations );
}

My current classes for this parsing situation

public class CreatedBy
{
    [JsonPropertyName( "displayName" )]
    public string DisplayName { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "id" )]
    public Guid Id { get; set; }
    [JsonPropertyName( "uniqueName" )]
    public string UniqueName { get; set; }
    [JsonPropertyName( "imageUrl" )]
    public string ImageUrl { get; set; }
}

public class PolicyConfigurationScope
{
    [JsonPropertyName( "refName" )]
    public string RefName { get; set; }
    [JsonPropertyName( "matchKind" )]
    public string MatchKind { get; set; }
    [JsonPropertyName( "repositoryId" )]
    public Guid RepositoryId { get; set; }
}

public class PolicyConfigurationSettings_MinimumNumberOfReviewers
{
    [JsonPropertyName( "minimumApproverCount" )]
    public int MinimumApproverCount { get; set; }
    [JsonPropertyName( "creatorVoteCounts" )]
    public bool CreatorVoteCounts { get; set; }
    [JsonPropertyName( "allowDownvotes" )]
    public bool AllowDownvotes { get; set; }
    [JsonPropertyName( "resetOnSourcePush" )]
    public bool ResetOnSourcePush { get; set; }
    [JsonPropertyName( "scope" )]
    public List<PolicyConfigurationScope> Scope { get; set; }
}

public class PolicyConfigurationType
{
    [JsonPropertyName( "id" )]
    public Guid Id { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "displayName" )]
    public string DisplayName { get; set; }
}

public class PolicyConfiguration
{
    [JsonPropertyName( "createdBy" )]
    public CreatedBy CreatedBy { get; set; }
    [JsonPropertyName( "createdDate" )]
    public DateTime CreatedDate { get; set; }
    [JsonPropertyName( "isEnabled" )]
    public bool IsEnabled { get; set; }
    [JsonPropertyName( "isBlocking" )]
    public bool IsBlocking { get; set; }
    [JsonPropertyName( "isDeleted" )]
    public bool IsDeleted { get; set; }
    //[JsonPropertyName( "settings" )]
    //public PolicyConfigurationSettings_MinimumNumberOfReviewersSettings Settings { get; set; }
    [JsonPropertyName( "revision" )]
    public int Revision { get; set; }
    [JsonPropertyName( "id" )]
    public int Id { get; set; }
    [JsonPropertyName( "url" )]
    public string Url { get; set; }
    [JsonPropertyName( "type" )]
    public PolicyConfigurationType Type { get; set; }
}

Broadminded

settings is a single token in your sample, why do you serialize to the List<T>? Are you parsing the scope values? Which parameter did you pass, which typeparams are used as T? – Serrate

@PavelAnikhouski That is my method to serialize the entire json result. In this case the result is a count of 4 and the parent object is an array of value types. settings is a child object of value, and is dynamic. I can post my object types if that helps? Essentially this is a PolicyConfiguration, and the property is PolicyConfigurationSettings. – Broadminded

@PavelAnikhouski I have updated my question with the classes used for deserializing. Settings is commented out and everything works, when I plug in the class for the listed setting, it attempts to parse Settings as that for every configuration, from here is where I am unsure where to go with a Converter. – Broadminded

--Edits: I believe I have everything in the original question now to address my current setup. – Broadminded

Thanks, the question is more clear now. Do you need an exact type for deserialization, have you considered to use a dictionary of keys and values? – Serrate

@PavelAnikhouski I'm honestly not sure what approach to take. This is my first time working with System.Text.Json, and even when using Newtonsoft, I hadn't encountered a complex object like this that I needed to work with. So any advice would be great. I can provide more settings examples if you would like. I appreciate your assistance! – Broadminded

Answer 1

I ended up solving my issue in slightly the same way I had seen a previous article using a discriminator. Since I do not control the API feeds, I do not have a discriminator to drive off of, so I am relying on the properties of the Json object.

Need to create a Converter:

public class PolicyConfigurationSettingsConverter : JsonConverter<PolicyConfigurationSettings>
{
    public override PolicyConfigurationSettings Read( ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options )
    {
        JsonDocument doc;
        JsonDocument.TryParseValue( ref reader, out doc );

        if ( doc.RootElement.TryGetProperty( "minimumApproverCount", out _ ) )
            return JsonSerializer.Deserialize<MinimumNumberOfReviewers>( doc.RootElement.ToString(), options );
        if ( doc.RootElement.TryGetProperty( "useSquashMerge", out _ ) )
            return JsonSerializer.Deserialize<RequireAMergeStrategy>( doc.RootElement.ToString(), options );
        if ( doc.RootElement.TryGetProperty( "scope", out _ ) )
            return JsonSerializer.Deserialize<PolicyConfigurationSettingsScope>( doc.RootElement.ToString(), options );

        return null;
    }

    public override void Write( Utf8JsonWriter writer, [DisallowNull] PolicyConfigurationSettings value, JsonSerializerOptions options )
    {
        if ( value.GetType() == typeof( MinimumNumberOfReviewers ) )
            JsonSerializer.Serialize( writer, ( MinimumNumberOfReviewers )value, options );
        if ( value.GetType() == typeof( RequireAMergeStrategy ) )
            JsonSerializer.Serialize( writer, ( RequireAMergeStrategy )value, options );
        if ( value.GetType() == typeof( PolicyConfigurationSettingsScope ) )
            JsonSerializer.Serialize( writer, ( PolicyConfigurationSettingsScope )value, options );
    }
}

Then need to create a JsonSerializerOptions object to add the Converter

public static JsonSerializerOptions PolicyConfigurationSettingsSerializerOptions()
{
    var serializeOptions = new JsonSerializerOptions();
    serializeOptions.Converters.Add( new PolicyConfigurationSettingsConverter() );
    return serializeOptions;
}

Pass the options into your Serializer/Deserializer statement.

Below is the PolicyConfigurationSettings class

public abstract class PolicyConfigurationSettings
{
    [JsonPropertyName( "scope" )]
    public List<PolicyConfigurationScope> Scope { get; set; }
}

public class MinimumNumberOfReviewers : PolicyConfigurationSettings
{
    [JsonPropertyName( "minimumApproverCount" )]
    public int MinimumApproverCount { get; set; }
    [JsonPropertyName( "creatorVoteCounts" )]
    public bool CreatorVoteCounts { get; set; }
    [JsonPropertyName( "allowDownvotes" )]
    public bool AllowDownvotes { get; set; }
    [JsonPropertyName( "resetOnSourcePush" )]
    public bool ResetOnSourcePush { get; set; }
}

public class RequireAMergeStrategy : PolicyConfigurationSettings
{
    [JsonPropertyName( "useSquashMerge" )]
    public bool UseSquashMerge { get; set; }
}

public class PolicyConfigurationSettingsScope : PolicyConfigurationSettings { }

Broadminded

JsonDocument implements IDisposable and in fact must be disposed because This class utilizes resources from pooled memory to minimize the impact of the garbage collector (GC) in high-usage scenarios. Failure to properly dispose this object will result in the memory not being returned to the pool, which will increase GC impact across various parts of the framework according to the docs. – Scamp

Also, you may get better performance re-serializing your JsonDocument to a byte array rather than a string for final deserialization. See System.Text.Json.JsonElement ToObject workaround. – Scamp

Thank you @dbc, I will make these changes to my code and update this answer! – Broadminded

Answer 2

In net 5.0 with System.Text.Json.JsonSerializer, what works for a class like this:

public class A
{
    public B Data { get; set; }
}
public class B
{
    public long Count { get; set; }
}

is using:

System.Text.Json.JsonSerializer.Deserialize<A>("{{\"data\":{\"count\":10}}}", new JsonSerializerOptions { PropertyNameCaseInsensitive = true, IncludeFields = true })

which is weird that is not the default.

Psoas

Answer 3

I solved this with a more generic approach, that falls somewhere between the way NewtonSoft Json and the .NET Json work. Using a custom converter, I serialize any polymorphic class, using a type identifier similar to the Newtonsoft approach, but to mitigate the possible security risk you can chose to allow only internal types or types from a specific assembly.

using System.Text.Json;
using System.Text.Json.Serialization;
using System.Diagnostics.CodeAnalysis;
using System.Diagnostics;
using System.Collections.ObjectModel;

public class JsonConverterEx<T> : System.Text.Json.Serialization.JsonConverter<T>
{
    private bool _internalOnly = true;
    private string _assembly = String.Empty;

    public JsonConverterEx()
    {
        this._assembly = this.GetType().Assembly.FullName;
    }

    public JsonConverterEx(bool bInternalOnly, string assemblyName)
    {
        _internalOnly = bInternalOnly;
        _assembly = assemblyName;
    }

    public override bool CanConvert(Type typeToConvert)
    {
        Type t = typeof(T);

        if(typeToConvert == t)
            return true;

        return false;
    }

    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        validateToken(reader, JsonTokenType.StartObject);

        reader.Read();      // Move to property name
        validateToken(reader, JsonTokenType.PropertyName);

        var typeKey = reader.GetString();

        reader.Read();      // Move to start of object (stored in this property)
        validateToken(reader, JsonTokenType.StartObject);

        if(!_internalOnly)
        {
            typeKey += ", " + _assembly;
        }

        Type t = Type.GetType(typeKey);
        if(t != null)
        {
            T o = (T)JsonSerializer.Deserialize(ref reader, t, options);
            reader.Read(); // Move past end of item object

            return o;
        }
        else
        {
            throw new JsonException($"Unknown type '{typeKey}'");
        }

        // Helper function for validating where you are in the JSON
        void validateToken(Utf8JsonReader reader, JsonTokenType tokenType)
        {
            if(reader.TokenType != tokenType)
                throw new JsonException($"Invalid token: Was expecting a '{tokenType}' token but received a '{reader.TokenType}' token");
        }
    }

    public override void Write(Utf8JsonWriter writer, [DisallowNull] T value, JsonSerializerOptions options)
    {
        var itemType = value.GetType();

        writer.WriteStartObject();
        writer.WritePropertyName(itemType.FullName);

        // pass on to default serializer
        JsonSerializer.Serialize(writer, value, itemType, options);

        writer.WriteEndObject();
    }
}

How to use it:

        JsonSerializerOptions op = new JsonSerializerOptions()
        {
            // your usual options here
        };
        op.Converters.Add(new JsonConverterEx<MyExternalClass>(false, "MyAssembly"));
        op.Converters.Add(new JsonConverterEx<MyInternalClass>());

        string s = System.Text.Json.JsonSerializer.Serialize(myobj, op);

        MyInternalClass c = System.Text.Json.JsonSerializer.Deserialize<MyInternalClass>(s, op);

Gameto

Answer 4

Alternatively, a more flexible design for serialization

public class PolymorphicConverter<T> : JsonConverter<T> where T : class
{
    public override T Read(
        ref Utf8JsonReader reader, 
        Type typeToConvert, 
        JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public override void Write(
        Utf8JsonWriter writer, 
        [DisallowNull] T value, 
        JsonSerializerOptions options)
    {
        JsonSerializer.Serialize(writer, value, value.GetType(), options);
    }
}

next, you can turn on your fantasy and customize deserialization

Do not forget:

options.JsonSerializerOptions.Converters.Add(new PolymorphicConverter<IFucker>());

Tennes

"Will of fantasy"? – Current

yes, sorry for wrong context, edited – Tennes