How to create an in-memory object that can be used as a Reader, Writer, or Seek in Rust?

Question

I need a completely in-memory object that I can give to BufReader and BufWriter. Something like Python's StringIO. I want to write to and read from such an object using methods ordinarily used with Files.

Is there a way to do this using the standard library?

Firecure

Answer 1

In fact there is a way: Cursor<T>!
(please also read Shepmaster's answer on why often it's even easier)

In the documentation you can see that there are the following impls:

impl<T> Seek for Cursor<T> where T: AsRef<[u8]>
impl<T> Read for Cursor<T> where T: AsRef<[u8]>
impl Write for Cursor<Vec<u8>>
impl<T> AsRef<[T]> for Vec<T>

From this you can see that you can use the type Cursor<Vec<u8>> just as an ordinary file, because Read, Write and Seek are implemented for that type!

Little example (Playground):

use std::io::{Cursor, Read, Seek, SeekFrom, Write};

// Create fake "file"
let mut c = Cursor::new(Vec::new());

// Write into the "file" and seek to the beginning
c.write_all(&[1, 2, 3, 4, 5]).unwrap();
c.seek(SeekFrom::Start(0)).unwrap();

// Read the "file's" contents into a vector
let mut out = Vec::new();
c.read_to_end(&mut out).unwrap();

println!("{:?}", out);

For a more useful example, check the documentation linked above.

Rhearheba

just wondering, would c.set_position(0) be equivalent to how you're seeking? – Boutique

Answer 2

You don't need a Cursor most of the time.

object that I can give to BufReader and BufWriter

BufReader requires a value that implements Read:

impl<R: Read> BufReader<R> {
    pub fn new(inner: R) -> BufReader<R>
}

BufWriter requires a value that implements Write:

impl<W: Write> BufWriter<W> {
    pub fn new(inner: W) -> BufWriter<W> {}
}

If you view the implementors of Read you will find impl<'a> Read for &'a [u8].

If you view the implementors of Write, you will find impl Write for Vec<u8>.

use std::io::{Read, Write};

fn main() {
    // Create fake "file"
    let mut file = Vec::new();

    // Write into the "file"
    file.write_all(&[1, 2, 3, 4, 5]).unwrap();

    // Read the "file's" contents into a new vector
    let mut out = Vec::new();
    let mut c = file.as_slice();
    c.read_to_end(&mut out).unwrap();

    println!("{:?}", out);
}

Writing to a Vec will always append to the end. We also take a slice to the Vec that we can update. Each read of c will advance the slice further and further until it is empty.

The main differences from Cursor:

• Cannot seek the data, so you cannot easily re-read data
• Cannot write to anywhere but the end

Patentor

This is great. Is it something we couldn't do over a year ago? – Firecure

@Firecure nope, those implementations existed in Rust 1.0. – Patentor

Answer 3

If you want to use BufReader with an in-memory String, you can use the as_bytes() method:

use std::io::BufRead;
use std::io::BufReader;
use std::io::Read;

fn read_buff<R: Read>(mut buffer: BufReader<R>) {
    let mut data = String::new();
    let _ = buffer.read_line(&mut data);

    println!("read_buff got {}", data);
}

fn main() {
    read_buff(BufReader::new("Potato!".as_bytes()));
}

This prints read_buff got Potato!. There is no need to use a cursor for this case.

To use an in-memory String with BufWriter, you can use the as_mut_vec method. Unfortunately it is unsafe and I have not found any other way. I don't like the Cursor approach since it consumes the vector and I have not found a way yet to use the Cursor together with BufWriter.

use std::io::BufWriter;
use std::io::Write;

pub fn write_something<W: Write>(mut buf: BufWriter<W>) {
    buf.write("potato".as_bytes());
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::io::{BufWriter};

    #[test]
    fn testing_bufwriter_and_string() {
        let mut s = String::new();

        write_something(unsafe { BufWriter::new(s.as_mut_vec()) });

        assert_eq!("potato", &s);
    }
}

Rectrix

You can get the Vec back after you are done with the Cursor. – Patentor

@Patentor Great! And then I can use from_utf8 to convet to String, from_utf8 seems to avoid copying of the vector so it should be efficient, could robably also use the `from_utf8_unchecked' which should be a simple move. – Rectrix