How can I distinguish between a deserialized field that is missing and one that is null?

Question

I'd like to use Serde to parse some JSON as part of a HTTP PATCH request. Since PATCH requests don't pass the entire object, only the relevant data to update, I need the ability to tell between a value that was not passed, a value that was explicitly set to null, and a value that is present.

I have a value object with multiple nullable fields:

struct Resource {
    a: Option<i32>,
    b: Option<i32>,
    c: Option<i32>,
}

If the client submits JSON like this:

{"a": 42, "b": null}

I'd like to change a to Some(42), b to None, and leave c unchanged.

I tried wrapping each field in one more level of Option:

#[derive(Debug, Deserialize)]
struct ResourcePatch {
    a: Option<Option<i32>>,
    b: Option<Option<i32>>,
    c: Option<Option<i32>>,
}

playground

This does not make a distinction between b and c; both are None but I'd have wanted b to be Some(None).

I'm not tied to this representation of nested Options; any solution that can distinguish the 3 cases would be fine, such as one using a custom enum.

Circosta

Answer 1

Quite likely, the only way to achieve that right now is with a custom deserialization function. Fortunately, it is not hard to implement, even to make it work for any kind of field:

fn deserialize_optional_field<'de, T, D>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
where
    D: Deserializer<'de>,
    T: Deserialize<'de>,
{
    Ok(Some(Option::deserialize(deserializer)?))
}

Then each field would be annotated as thus:

#[serde(deserialize_with = "deserialize_optional_field")]
a: Option<Option<i32>>,

You also need to annotate the struct with #[serde(default)], so that empty fields are deserialized to an "unwrapped" None. The trick is to wrap present values around Some.

Serialization relies on another trick: skipping serialization when the field is None:

#[serde(deserialize_with = "deserialize_optional_field")]
#[serde(skip_serializing_if = "Option::is_none")]
a: Option<Option<i32>>,

Playground with the full example. The output:

Original JSON: {"a": 42, "b": null}
> Resource { a: Some(Some(42)), b: Some(None), c: None }
< {"a":42,"b":null}

Loisloise

See also Support, or at least document the "double option" pattern – Circosta

I'm a bit flabbergasted that even to this day we don't have a simple directive to turn off deserialization for struct item or struct within a struct. The above code was an excellent start but doesn't work for structs within structs that return empty. Many Rest APIs have different items returning null or none depending on the request. With a lot of mucking about I've implemented my own function based on the above but it shouldn't be this hard? – Vladikavkaz

Answer 2

Building off of E_net4's answer, you can also create an enum for the three possibilities:

#[derive(Debug)]
enum Patch<T> {
    Missing,
    Null,
    Value(T),
}

impl<T> Default for Patch<T> {
    fn default() -> Self {
        Patch::Missing
    }
}

impl<T> From<Option<T>> for Patch<T> {
    fn from(opt: Option<T>) -> Patch<T> {
        match opt {
            Some(v) => Patch::Value(v),
            None => Patch::Null,
        }
    }
}

impl<'de, T> Deserialize<'de> for Patch<T>
where
    T: Deserialize<'de>,
{
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        Option::deserialize(deserializer).map(Into::into)
    }
}

This can then be used as:

#[derive(Debug, Deserialize)]
struct ResourcePatch {
    #[serde(default)]
    a: Patch<i32>,
}

Unfortunately, you still have to annotate each field with #[serde(default)] (or apply it to the entire struct). Ideally, the implementation of Deserialize for Patch would handle that completely, but I haven't figured out how to do that yet.

Circosta

"Ideally, the implementation of Deserialize for Patch would handle that completely" My assumption so far is that this is impossible. From the moment that you attempt to deserialize a Patch, you expect the value to exist in its serialized form. On the contrary, Patch::Missing exists through its "non-existence" in our container (you cannot actually serialize Patch::Missing to JSON on its own). AFAIK a Serialize cannot choose not to be serialized, nor can it tell the container to skip that part of the process. – Loisloise

Well, actually, my previous comment is more applicable to serialization, and not deserialization. Regardless, the logic is somewhat dual: Deserialize cannot tell the Deserializer to deserialize values that it just won't find. If we have an empty object {}, the implementations of Deserialize can do nothing about it, but the deserializer can, once it knows to fill in a default. – Loisloise

I prefer this approach to the "double option" pattern as it gives you a more explicit and understandable type than wrapped options do- thanks! Solves https://mcmap.net/q/330023/-how-can-i-distinguish-between-a-deserialized-field-that-is-missing-and-one-that-is-null/141881 for me :D – Kianakiang

Answer 3

Building up on Shepmaster's answer and adding serialization.

use serde::ser::Error;
use serde::{Deserialize, Deserializer};
use serde::{Serialize, Serializer};

// #region ------ JSON Absent support
// build up on top of https://stackoverflow.com/a/44332837

/// serde Valueue that can be Absent, Null, or Valueue(T)
#[derive(Debug)]
pub enum Maybe<T> {
    Absent,
    Null,
    Value(T),
}

#[allow(dead_code)]
impl<T> Maybe<T> {
    pub fn is_absent(&self) -> bool {
        match &self {
            Maybe::Absent => true,
            _ => false,
        }
    }
}

impl<T> Default for Maybe<T> {
    fn default() -> Self {
        Maybe::Absent
    }
}

impl<T> From<Option<T>> for Maybe<T> {
    fn from(opt: Option<T>) -> Maybe<T> {
        match opt {
            Some(v) => Maybe::Value(v),
            None => Maybe::Null,
        }
    }
}

impl<'de, T> Deserialize<'de> for Maybe<T>
where
    T: Deserialize<'de>,
{
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        Option::deserialize(deserializer).map(Into::into)
    }
}

impl<T: Serialize> Serialize for Maybe<T> {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        match self {
            // this will be serialized as null
            Maybe::Null => serializer.serialize_none(),
            Maybe::Value(v) => v.serialize(serializer),
            // should have been skipped
            Maybe::Absent => Err(Error::custom(
                r#"Maybe fields need to be annotated with: 
  #[serde(default, skip_serializing_if = "Maybe::is_Absent")]"#,
            )),
        }
    }
}
// #endregion --- JSON Absent support

And then you can use it this way:

#[derive(Serialize, Deserialize, Debug)]
struct Rect {
    #[serde(default, skip_serializing_if = "Maybe::is_absent")]
    stroke: Maybe<i32>,

    w: i32,

    #[serde(default, skip_serializing_if = "Maybe::is_absent")]
    h: Maybe<i32>,
}


// .... 


let json = r#"
{
  "stroke": null,
  "w": 1
}"#;
    
let deserialized: Rect = serde_json::from_str(json).unwrap();
println!("deserialized = {:?}", deserialized);
// will output: Rect { stroke: Null, w: 1, h: Absent }

let serialized = serde_json::to_string(&deserialized).unwrap();
println!("serialized back = {}", serialized);
// will output: {"stroke":null,"w":1}

I wish Serde had a built-in way to handle JSON's null and absent states.

Update 2021-03-12 - Updated to Maybe::Absent as it is more JSON and SQL DSL idiomatic.

The catch with this approach is that we can express:

• type | null with the default Option<type>
• type | null | absent with Maybe<type>

But we cannot express

• type | absent

The solution would be to refactor Maybe to just have ::Present(value) and ::Absent and support Maybe<Option<type>> for the type | null | absent. So this will give us full coverage.

• type | null with the default Option<type>
• type | absent with Maybe<type>
• type | absent | null with Maybe<Option<type>>

I am trying to implement this without adding a #[serde(deserialize_with = "deserialize_maybe_field")] but not sure it is possible. I might be missing something obvious.

Margartmargate

Answer 4

You can now use double_option from the serde_with crate, which does exactly what you want.

Educt