How to deserialize a map into a vector of a custom struct, with a field matching the key?

Question

Given a yaml with a map of items,

items:
  item1:
    uid: ab1234
    foo: bar
  item2:
    uid: cd5678
    foo: baz

how can I parse it with serde into a Vec<Item> with a new field, "name", generated from the key of the original map,

struct Item {
  name: String,  // to be populated by the key (item1, item2)
  uid: String,
  foo: String,
}

I'm trying to get the following code (rustexplorer) to work, which currently errors out with Err(Error("items: invalid type: map, expected a sequence", line: 3, column: 3)).

/*
[dependencies]
serde = "1.0.193"
serde_derive = "1.0.193"
serde_yaml = "0.9.27"
*/

use serde_derive::{Deserialize, Serialize};
use std::vec::Vec;


#[derive(Debug, Serialize, Deserialize)]
struct Item {
  name: String,
  uid: String,
  foo: String,
}

#[derive(Debug, Serialize, Deserialize)]
struct Schema {
    items: Vec<Item>,
}

fn main() {
    let input = r###"
items:
  item1:
    uid: ab1234
    foo: bar
  item2:
    uid: cd5678
    foo: baz
"###;

    let items = serde_yaml::from_str::<Schema>(input);
    println!("{:?}", items);
}

Icebreaker

Seems doable with serde_as but not sure how. – Icebreaker

ot: You can import serde_derive through the main serde crate with the feature derive and replace serde_derive with just serde – Essive

Answer 1

serde_with has something for exactly that: KeyValueMap:

use serde::{Deserialize, Serialize};
use serde_with::{serde_as, KeyValueMap};

#[derive(Debug, Serialize, Deserialize)]
struct Item {
    #[serde(rename = "$key$")]
    name: String,
    uid: String,
    foo: String,
}

#[serde_as]
#[derive(Debug, Serialize, Deserialize)]
struct Schema {
    #[serde_as(as = "KeyValueMap<_>")]
    items: Vec<Item>,
}

Quesnay

Answer 2

You can do it without intermediate allocations using a custom deserializer and a Visitor:

use serde::Deserializer;
use serde::de::MapAccess;
use serde::de::Visitor;
use serde_derive::{Deserialize, Serialize};
use std::fmt;
use std::vec::Vec;


#[derive(Debug, Serialize, Deserialize)]
struct Item {
  name: String,
  uid: String,
  foo: String,
}

#[derive(Debug, Serialize, Deserialize)]
struct Schema {
    #[serde(deserialize_with = "deserialize_items")]
    items: Vec<Item>,
}

fn deserialize_items<'de, D> (deserializer: D) -> Result<Vec<Item>, D::Error>
where D: Deserializer<'de>,
{
    struct ItemsVisitor;

    impl<'de> Visitor<'de> for ItemsVisitor
    {
        type Value = Vec<Item>;

        fn expecting (&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str("an items map")
        }
        
        fn visit_map<M> (self, mut map: M) -> Result<Vec<Item>, M::Error>
        where M: MapAccess<'de>,
        {
            #[derive(Deserialize)]
            struct ItemFields {
                uid: String,
                foo: String,
            }

            let mut result = vec![];
            while let Some ((name, fields)) = map.next_entry::<_, ItemFields>()? {
                result.push (Item { name, uid: fields.uid, foo: fields.foo });
            }
            Ok (result)
        }
    }
    
    deserializer.deserialize_map (ItemsVisitor)
}

fn main() {
    let input = r###"
items:
  item1:
    uid: ab1234
    foo: bar
  item2:
    uid: cd5678
    foo: baz
"###;

    let items = serde_yaml::from_str::<Schema>(input);
    println!("{:?}", items);
}

Playground

See also: https://serde.rs/stream-array.html

Surprise

Answer 3

While certainly not the cleanest solution to this problem, you could combine the tuple_vec_map crate and a custom deserialization function:

#[derive(Debug, Serialize, Deserialize)]
struct Schema {
    #[serde(deserialize_with = "vec_item")]
    items: Vec<Item>,
}

fn vec_item<'de, D>(de: D) -> Result<Vec<Item>, D::Error>
where
    D: Deserializer<'de>,
{
    #[derive(Deserialize)]
    struct ItemSerde {
        uid: String,
        foo: String,
    }
    #[derive(Deserialize)]
    #[serde(transparent)]
    struct SchemaSerde(#[serde(with = "tuple_vec_map")] Vec<(String, ItemSerde)>);

    let SchemaSerde(items) = Deserialize::deserialize(de)?;
    Ok(items
        .into_iter()
        .map(|(name, ItemSerde { uid, foo })| Item { name, uid, foo })
        .collect())
}

Explorer

Lakin

Answer 4

You could read into a temporary map of String -> RawItem (item without name), and transform that into the structure you need:

#[derive(Debug, Deserialize)]
struct Schema {
    #[serde(deserialize_with = "deserialize_schema_items")]
    items: Vec<Item>,
}

fn deserialize_schema_items<'de, D: Deserializer<'de>>(d: D) -> Result<Vec<Item>, D::Error> {
    #[derive(Deserialize)]
    struct RawItem {
        uid: String,
        foo: String,
    }
    let raw_items: IndexMap<String, RawItem> = Deserialize::deserialize(d)?;
    let items = raw_items
        .into_iter()
        .map(|(name, raw_item)| Item {
            name,
            uid: raw_item.uid,
            foo: raw_item.foo,
        })
        .collect();
    Ok(items)
}

The above code uses indexmap::IndexMap to preserve order. If you don't care about order, you can use HashMap instead. If you opt for IndexMap, note that you'll need to turn on the serde feature of the indexmap crate.

Piroshki