How do I use Serde to deserialize structs with references from a reader?

Question

I have these structs:

#[derive(Debug, Serialize, Deserialize)]
pub struct GGConf<'a> {
    #[serde(alias = "ssh")]
    #[serde(rename = "ssh")]
    #[serde(default)]
    #[serde(borrow)]
    pub ssh_config: Option<SSHConfig<'a>>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct SSHConfig<'a> {
    #[serde(alias = "privateKey")]
    #[serde(rename = "privateKey")]
    private_key: &'a str,

    username: &'a str,
}

Deserialization happens when I read from a YAML file:

let mut config: GGConf = serde_yaml::from_reader(file)?;

On compiling, I get an error:

error: implementation of `conf::_IMPL_DESERIALIZE_FOR_GGConf::_serde::Deserialize` is not general enough
   --> src/conf.rs:50:34
    |
50  |           let mut config: GGConf = serde_yaml::from_reader(file)?;
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^ implementation of `conf::_IMPL_DESERIALIZE_FOR_GGConf::_serde::Deserialize` is not general enough
    |
   ::: /home/ninan/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.98/src/de/mod.rs:524:1
    |
524 | / pub trait Deserialize<'de>: Sized {
525 | |     /// Deserialize this value from the given Serde deserializer.
526 | |     ///
527 | |     /// See the [Implementing `Deserialize`][impl-deserialize] section of the
...   |
562 | |     }
563 | | }
    | |_- trait `conf::_IMPL_DESERIALIZE_FOR_GGConf::_serde::Deserialize` defined here
    |
    = note: `conf::GGConf<'_>` must implement `conf::_IMPL_DESERIALIZE_FOR_GGConf::_serde::Deserialize<'0>`, for any lifetime `'0`...
    = note: ...but `conf::GGConf<'_>` actually implements `conf::_IMPL_DESERIALIZE_FOR_GGConf::_serde::Deserialize<'1>`, for some specific lifetime `'1`

I vaguely understand that serde deserialization also has a lifetime 'de and that the compiler is confusing my lifetime specified for it? Please correct me if I'm wrong.

How do I currently correctly deserialize the YAML into both structs? Is there something I am missing here or misunderstood?

I looked at How do I resolve "implementation of serde::Deserialize is not general enough" with actix-web's Json type?, but I cannot use an owned type. I need it to be a borrowed type.

I will try and write a playground example for this.

Furmenty

It's hard to answer your question because it doesn't include a minimal reproducible example. We can't tell what crates (and their versions), types, traits, fields, etc. are present in the code. It would make it easier for us to help you if you try to reproduce your error on the Rust Playground if possible, otherwise in a brand new Cargo project, then edit your question to include the additional info. There are Rust-specific MRE tips you can use to reduce your original code for posting here. Thanks! – Kandace

Answer 1

This is impossible; you must use owned data instead of references.

Here's a minimal example:

use serde::Deserialize; // 1.0.104

#[derive(Debug, Deserialize)]
pub struct SshConfig<'a> {
    username: &'a str,
}

fn example(file: impl std::io::Read) {
    serde_yaml::from_reader::<_, SshConfig>(file);
}

error: implementation of `_IMPL_DESERIALIZE_FOR_SshConfig::_serde::Deserialize` is not general enough
   --> src/lib.rs:9:5
    |
9   |       serde_yaml::from_reader::<_, SshConfig>(file);
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ implementation of `_IMPL_DESERIALIZE_FOR_SshConfig::_serde::Deserialize` is not general enough
    | 
   ::: /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/src/de/mod.rs:531:1
    |
531 | / pub trait Deserialize<'de>: Sized {
532 | |     /// Deserialize this value from the given Serde deserializer.
533 | |     ///
534 | |     /// See the [Implementing `Deserialize`][impl-deserialize] section of the
...   |
569 | |     }
570 | | }
    | |_- trait `_IMPL_DESERIALIZE_FOR_SshConfig::_serde::Deserialize` defined here
    |
    = note: `SshConfig<'_>` must implement `_IMPL_DESERIALIZE_FOR_SshConfig::_serde::Deserialize<'0>`, for any lifetime `'0`...
    = note: ...but `SshConfig<'_>` actually implements `_IMPL_DESERIALIZE_FOR_SshConfig::_serde::Deserialize<'1>`, for some specific lifetime `'1`

If you look at the definition of serde_yaml::from_reader, you'll see that it's limited to only deserializing owned data:

pub fn from_reader<R, T>(rdr: R) -> Result<T>
where
    R: Read,
    T: DeserializeOwned,
//     ^^^^^^^^^^^^^^^^ 

The same is true for serde_json::from_reader and probably any equivalent function.

You can only deserialize a type containing references when there's data to reference. Something implementing the Read trait only guarantees that it can copy some bytes into a user-provided buffer. Since the from_reader function doesn't accept that buffer as an argument, any buffer would be destroyed at the exit of from_reader, invalidating the references.

See also:

• How do I resolve "implementation of serde::Deserialize is not general enough" with actix-web's Json type?

---

If you must use references (and in many cases this isn't true), you will need to:

1. read from the reader yourself into a buffer
2. use from_str instead of from_reader
3. keep the buffer around as long as the deserialized data

Kandace

Will try out the suggestion. I need the SSHConfig values to be references as I will be overwriting a static constant with the value from the file. Thus forcing me to have references. – Furmenty

@Furmenty Sounds like you want How do I create a global, mutable singleton?. You won't be able to directly use references loaded from a file in your case as they won't be 'static anyway. – Kandace

Yes. I have a global mutable singleton with a default SSHConfig value which may or may not be overridden from the file. – Furmenty

Global mutable singletons can have Strings. I suspect you are suffering from the XY problem. – Kandace

Can this be achieved without lazy_static? I just removed all references and I get this error on creation of the singleton -> error[E0015]: calls in statics are limited to constant functions, tuple structs and tuple variants. I am ok with using lazy_static. Just asking to know what is different. – Furmenty

@Furmenty yes it can be done without lazy_static, but no, it cannot be done without something like lazy_static. The singleton link from earlier shows alternatives. Chances are, your attempt at using a static would introduce undefined behavior. – Kandace

One point I expanded on in my answer - serde_yaml doesn't support zero-copy deserialization currently (as of v0.8), so the from_str method still requires owned data types – Percheron

Answer 2

from_reader takes a stream of data from somewhere (anywhere which implements the Read trait) - it doesn't store the data, meaning nothing owns the data, therefore you cannot have a reference to that data in your struct. In other words, from_reader takes a transient stream of data, therefore needs a place to store the data.

An additional complication is that serde_yaml (at least for version 0.8.11) doesn't not support zero-copy deserialization:

https://docs.rs/serde_yaml/0.8.11/serde_yaml/fn.from_str.html

pub fn from_str<T>(s: &str) -> Result<T> where
    T: DeserializeOwned,

...

YAML currently does not support zero-copy deserialization.

Compare this to, say, serde_json, which does:

https://docs.rs/serde_json/1.0.50/serde_json/de/fn.from_str.html

pub fn from_str<'a, T>(s: &'a str) -> Result<T> where
    T: Deserialize<'a>,

So, at least with something like serde_json you could use from_str from an owned buffer, and that will allow you to use references in your struct (but this will not work for serde_yaml currently)

// Written with rustc 1.42.0 and
// [dependencies]
// serde = "1.0.105"
// serde_derive = "1.0.105"
// serde_json = "1.0.50"

use std::io::Read;
use serde_derive::Deserialize;

#[derive(Debug, Deserialize)]
pub struct SshConfig<'a> {
    username: &'a str,
}

fn main() {
    // Open file handle
    let mut file = std::fs::File::open("example.json").unwrap();

    // Read the data into a String, which stores (and thus owns) the data
    let mut strbuf = String::new();
    file.read_to_string(&mut strbuf).unwrap();

    // Deserialize into struct, which references
    let result: SshConfig = serde_json::from_str(&strbuf).unwrap();
    println!("{:?}", result.username);

    // Note that `result` is only valid as long as `strbuf` exists.
    // i.e if `strbuf` goes out of scope or is moved to another function, we get an error. For example, the following would cause an error:
    // std::mem::drop(strbuf); // Function which moves strbuf, not a referernce
    // println!("{:?}", result.username); // Error
}

Depending on exactly what your concerns are, this might be less efficient than storing a String in your struct (e.g if the example.json is 1MB large, and you only extract a single field - the above code will store the entire 1MB string in memory only to have a few bytes worth of text accessible).

Percheron