Custom JSON serialization of sensitive data for PCI compliance

Question

We have to log incoming requests and outgoing responses for our web service. This includes JSON serialization of each object so they can be stored in a database.

Some information is considered sensitive (such as Social Security Numbers, credit card numbers, etc.) and we cannot include these in our logs per PCI compliance. Right now we're manually replacing the values with a placeholder value (e.g. "[PRIVATE]") but this only works with string properties. Some data, such as a Date of Birth is not stored as a string so this doesn't work as the replacement of the property value happens before the serialization. The big problem is that it is too easy for someone to forget to do remove the sensitive data before logging it, which is highly undesirable.

To remedy this, I was thinking of creating a custom attribute and placing it on the property and then having the JSON serialization routine look for this attribute on each property and if it exists, replace the serialized value with a placeholder such as "[PRIVATE]".

Right now we are using the System.Web.Script.Serialization.JavaScriptSerializer for our serialization. Obviously it knows nothing of my custom attribute. How would I go about changing the serialization process so any data decorated with my custom "SensitiveData" attribute is replaced with a placeholder value? I'm not against using a different serializer but was hoping I could leverage the features of an existing one instead of writing my own.

Augustinaaugustine

Use Newtonsoft JSON, see this q/a: https://mcmap.net/q/867121/-add-a-custom-attribute-to-json-net/1300910 – Overrule

JavaScriptSerializer is very bare-bones. The only way to do something like this is by writing a generic JavaScriptConverter that you use for all of your types that manually iterates the properties using reflection and skips ones with the marked attribute. Better to use json.net. – Palecek

However, if you do switch to Json.NET, you can use a custom contract resolver along the lines of the one shown in How can I encrypt selected properties when serializing my objects?. You'd need to tweak it to your needs however since that contract resolver only works for string-valued properties. Another approach is shown in JSON.Net custom contract serialization and Collections. So, are you willing to switch to json.net? If so we could provide a more specific answer than those. – Palecek

I have switched to JSON.NET and created a new class which inherits from JsonConverter. Since I'm only concerned with serialization, I overrode the WriteJson method to loop through the "value" object's properties and see if my SenstiveData attribute is applied. If it is, I replace the property value with my placeholder otherwise I just serialize it normally. So far this seems to work with an object with simple properties but I need to make sure it works with objects with complex properties as well as value type variables. If it looks good, I'll post my solution here. – Augustinaaugustine

Answer 1

Here's my solution, although it may need minor tweaks:

My custom JsonConverter:

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
using System.Linq;
using System.Reflection;

public class SensitiveDataJsonConverter : JsonConverter
{
    private readonly Type[] _types;

    public SensitiveDataJsonConverter(params Type[] types)
    {
      _types = types;
    }

    public override bool CanConvert(Type objectType)
    {
        return _types.Any(e => e == objectType);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var jObject = new JObject();
        var type = value.GetType();

        foreach (var propertyInfo in type.GetProperties())
        {
            // We can only serialize properties with getters
            if (propertyInfo.CanRead)
            {
                var sensitiveDataAttribute = propertyInfo.GetCustomAttribute<SensitiveDataAttribute>();
                object propertyValue;

                if (sensitiveDataAttribute != null)
                    propertyValue = "[REDACTED]";
                else
                    propertyValue = propertyInfo.GetValue(value);

                if (propertyValue == null)
                  propertyValue = string.Empty;

                var jToken = JToken.FromObject(propertyValue, serializer);

                jObject.Add(propertyInfo.Name, jToken);
            }
        }

        jObject.WriteTo(writer);
    }

Here's my custom attribute:

using System;

[AttributeUsage(AttributeTargets.Parameter | AttributeTargets.Property)]
public class SensitiveDataAttribute : Attribute
{

}

We use it like this:

string serialized;

try
{
    serialized = JsonConvert.SerializeObject(value, new SensitiveDataJsonConverter(value.GetType()));
}
catch // Some objects cannot be serialized
{
    serialized = $"[Unable to serialize object '{key}']";
}

Here's a test class I try to serialize:

class Person
{
    public Person()
    {
        Children = new List<Person>();
    }

    public List<Person> Children { get; set; }

    [SensitiveData]
    public DateTime DateOfBirth { get; set; }

    public string FirstName { get; set; }

    public string LastName { get; set; }

    [SensitiveData]
    public string SocialSecurityNumber { get; set; }

    public Person Spouse { get; set; }
}

This seemed to work great until I added the Spouse and Children properties but I was getting a NullReferenceException. I added this to the WriteJson method which corrected the problem:

if (propertyValue == null)
    propertyValue = string.Empty;

Augustinaaugustine

This was super useful!! The only tweak that I would suggest is that instead of the if (propertyValue == null) check to replace it with string.Empty, you could do JValue.CreateNull() instead. This keeps the value as null in the final serialized output instead of technically changing its value to be empty string instead. Here's how I implemented it in mine. var jToken = propertyValue == null ? JValue.CreateNull() : JToken.FromObject(propertyValue, serializer); – Fugacious

Answer 2

I was also looking for a way to hide GDPR data in Audit information. So i came across this question.

I've had a lot of issues implementing this JsonConvertor. I had a lot of issues with types and certainly with child types (i was serializing the AuditEvent from Audit.NET lib, and then some child props of that). I even added the code from @Duu82, but then still a lot of issues.

I however found another way of resolving my problem, and it was in another question/answer on SO: Replace sensitive data value on JSON serialization

So for future users who come across this, I found that answer more practical, less code and working

Instil

Answer 3

@DesertFoxAZ had a great answer in here. To augment that answer, I would add in the below code. Without this code, the PII redaction will only apply to the top-level class that you pass into your constructor. If you have properties that are objects that should also have some of their data redacted, you need the below code as well.

The code below was modified from an answer in this SO post.

        private readonly List<Type> _alreadyVisitedTypes = new List<Type>(); //avoid infinite recursion

        private void RecursivelySetTypesToOperateOn(Type currentType)
        {
            if (_alreadyVisitedTypes.Contains(currentType))
            {
                return;
            }
            _alreadyVisitedTypes.Add(currentType);

            if (currentType.IsClass && currentType.Namespace != "System") //custom defined classes only
            {
                _types.Add(currentType);
            }
            foreach (PropertyInfo pi in currentType.GetProperties())
            {
                if (pi.PropertyType.IsClass)
                {
                    RecursivelySetTypesToOperateOn(pi.PropertyType);
                }
            }
        }

Fugacious

Answer 4

I also wrote a recursive version of @DesertFoxAZ's answer for multi-level objects.

public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
    var writingObject = RedactSensitiveProperties(value, serializer);
    writingObject.WriteTo(writer);
}

private static JObject RedactSensitiveProperties(object value, JsonSerializer serializer)
{
    var writingObject = new JObject();
    var type = value.GetType();

    foreach (var propertyInfo in type.GetProperties())
    {
        // We can only serialize properties with getters
        if (propertyInfo.CanRead)
        {
            var sensitiveDataAttribute = propertyInfo.GetCustomAttribute<SensitiveDataAttribute>();
            object propertyValue;

            if (sensitiveDataAttribute != null)
            {
                propertyValue = "[REDACTED]";
            }
            else
            {
                propertyValue = propertyInfo.GetValue(value);
            }

            if (propertyValue == null)
            {
                propertyValue = string.Empty;
            }

            var jToken = JToken.FromObject(propertyValue, serializer);

            if (jToken.Type == JTokenType.Object)
            {
                var jObject = RedactSensitiveProperties(propertyValue, serializer);
                writingObject.Add(propertyInfo.Name, jObject);
            }
            else if (jToken.Type == JTokenType.Array)
            {
                var jArray = new JArray();
                foreach (var o in (IEnumerable<object>)propertyValue)
                {
                    jArray.Add(RedactSensitiveProperties(o, serializer));
                }
                writingObject.Add(propertyInfo.Name, jArray);
            }
            else
            {
                writingObject.Add(propertyInfo.Name, jToken);
            }
        }
    }

    return writingObject;
}

Scorpaenid