How do I use Serde to (de)serialize arrays greater than 32 elements, such as [u8; 128]?

Question

I have a struct containing a byte array that I would like to serialize and deserialize to and from binary, but it only works for arrays up to 32 elements.

Here is my minimal example code

main.rs:

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate bincode;

use bincode::{serialize, deserialize, Infinite};

const BYTECOUNT: usize = 32; // 33 and more does not work, I need 128
type DataArr = [u8; BYTECOUNT];

#[derive(Serialize, Deserialize, Debug)]
struct Entry {
    number: i64,
    data: DataArr
}

fn main() {
    let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] };
    my_entry.data[4] = 42;

    // Convert the Entry to binary.
    let serialized: Vec<u8> = serialize(&my_entry, Infinite).unwrap();
    println!("serialized = {:?}", serialized);

    // Convert the binary representation back to an Entry.
    let deserialized: Entry = deserialize(&serialized).unwrap();
    println!("deserialized = {:?}", deserialized);
}

Cargo.toml:

[package]
name = "array_serialization_test"
version = "0.1.0"

[dependencies]
serde = "*"
serde_derive = "*"
bincode = "*"

output:

serialized = [57, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
deserialized = Entry { number: 12345, data: [0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 

How can I make it work for 128 elements in the array? Can I somehow manually extend array_impls! in my user code? Is there an alternative approach?

I think this question is different from How do I map a C struct with padding over 32 bytes using serde and bincode? because I actually need the content of the array, since it is not just used for padding. Also I would like to know if I can extend array_impls! on my code.

Stopping

See also #44468886. – Faraway

Answer 1

Currently there is no way for Serde to provide Serialize and Deserialize impls that work for every array size. This is blocked on const generics which is being worked on and will hopefully land in nightly later in 2018.

For now you can define your own "big array" helper that can serialize and deserialize arrays of any specific sizes used in your crate. Fields for which you want to use the big array helper will need to be tagged with #[serde(with = "BigArray")] or else Serde will look for non-existent Serialize and Deserialize impls.

#[macro_use]
extern crate serde_derive;

extern crate serde;
extern crate bincode;

mod big_array;
use big_array::BigArray;

const BYTECOUNT: usize = 128;
type DataArr = [u8; BYTECOUNT];

#[derive(Serialize, Deserialize)]
struct Entry {
    number: i64,
    #[serde(with = "BigArray")]
    data: DataArr
}

fn main() {
    let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] };
    my_entry.data[4] = 42;

    // Convert the Entry to binary.
    let serialized: Vec<u8> = bincode::serialize(&my_entry).unwrap();
    println!("serialized = {:?}", serialized);

    // Convert the binary representation back to an Entry.
    let deserialized: Entry = bincode::deserialize(&serialized).unwrap();
    println!("deserialized = {} {:?}", deserialized.number, &deserialized.data[..]);
}

The big array helper can be defined in src/big_array.rs as follows. Maybe this would make a good crate by itself if you would like to own it!

use std::fmt;
use std::marker::PhantomData;
use serde::ser::{Serialize, Serializer, SerializeTuple};
use serde::de::{Deserialize, Deserializer, Visitor, SeqAccess, Error};

pub trait BigArray<'de>: Sized {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where S: Serializer;
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
        where D: Deserializer<'de>;
}

macro_rules! big_array {
    ($($len:expr,)+) => {
        $(
            impl<'de, T> BigArray<'de> for [T; $len]
                where T: Default + Copy + Serialize + Deserialize<'de>
            {
                fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
                    where S: Serializer
                {
                    let mut seq = serializer.serialize_tuple(self.len())?;
                    for elem in &self[..] {
                        seq.serialize_element(elem)?;
                    }
                    seq.end()
                }

                fn deserialize<D>(deserializer: D) -> Result<[T; $len], D::Error>
                    where D: Deserializer<'de>
                {
                    struct ArrayVisitor<T> {
                        element: PhantomData<T>,
                    }

                    impl<'de, T> Visitor<'de> for ArrayVisitor<T>
                        where T: Default + Copy + Deserialize<'de>
                    {
                        type Value = [T; $len];

                        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                            formatter.write_str(concat!("an array of length ", $len))
                        }

                        fn visit_seq<A>(self, mut seq: A) -> Result<[T; $len], A::Error>
                            where A: SeqAccess<'de>
                        {
                            let mut arr = [T::default(); $len];
                            for i in 0..$len {
                                arr[i] = seq.next_element()?
                                    .ok_or_else(|| Error::invalid_length(i, &self))?;
                            }
                            Ok(arr)
                        }
                    }

                    let visitor = ArrayVisitor { element: PhantomData };
                    deserializer.deserialize_tuple($len, visitor)
                }
            }
        )+
    }
}

big_array! {
    40, 48, 50, 56, 64, 72, 96, 100, 128, 160, 192, 200, 224, 256, 384, 512,
    768, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
}

Canossa

Could you explain how come this solution doesn't suffer from the error: type parameter ``T`` must be used as the type parameter for some local type? Because T could be anything here, so I thought for [T; $len] should fail (it doesn't fail, the solution is valid) – Ashia

for ... { seq.serialize_element ... } -> Will it be properly inlined and optimized, or do something byte by byte? Can it be somehow done with serialize_bytes? – Tomboy

You can now define BigArray with const generics to support any size. – Radiant

Answer 2

As of 2023, serde still doesn't support generic-length arrays since this would break the impl for [T; 0]. However, you can still take advantage of const generics to create a fully generic solution yourself.

Here is a serializer function.

use serde::ser::SerializeTuple;
use serde::{Serialize, Serializer};

fn serialize<const N: usize, S, T>(t: &[T; N], serializer: S) -> Result<S::Ok, S::Error>
where
    S: Serializer,
    T: Serialize,
{
    let mut ser_tuple = serializer.serialize_tuple(N)?;
    for elem in t {
        ser_tuple.serialize_element(elem)?;
    }
    ser_tuple.end()
}

You can then use #[serde(serialize_with = "serialize")]

const BYTECOUNT: usize = 128;
type DataArr = [u8; BYTECOUNT];

#[derive(Serialize, Debug)]
pub struct Entry {
    number: i64,
    #[serde(serialize_with = "serialize")]
    data: DataArr,
}

It is also possible to make the deserializer, but that is much more complex due to having a partially initialized array, so instead I'll skip to the real solution: use serde_with, which can handle arbitrarily large arrays and nested arrays. In general #[serde_as(as = "[_; BYTECOUNT]")] should work for arrays containing any type, while this case specifically can use Bytes.

use serde_with::{serde_as, Bytes};
use serde::{Serialize, Deserialize};

const BYTECOUNT: usize = 128;
type DataArr = [u8; BYTECOUNT];

#[serde_as]
#[derive(Serialize, Deserialize, Debug)]
pub struct Entry {
    number: i64,
    #[serde_as(as = "Bytes")]
    data: DataArr,
}

Here is serde_with's deserializer in case you are curious or want to write it yourself.

Brokaw

Answer 3

This question comes up a lot in google searches about (de)serializing arrays of const generic size, so I am adding a solution that is relevant for that as well, as it is also relevant to this question.
There are now the crates serde_with and serde_arrays that can help serde (de)serialize arrays of any size.

For example: use serde_with by adding it to Cargo.toml and decorating the struct like so:

use serde::{Serialize, Deserialize};
use serde_with::serde_as;

#[serde_as]
#[derive(Serialize, Deserialize)]
struct GenericArray<const N: usize> {
    number: i64,
    #[serde_as(as = "[_; N]")]
    data: [u8; N],
}

Or add serde_arrays to Cargo.toml and use like so:

use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct GenericArray<const N: usize> {
    number: i64,
    #[serde(with = "serde_arrays")]
    data: [u8, N],
}

Reformation

Answer 4

I think this is just waiting on const generics support in Serde.

Toscana