Reading specific number of bytes from a buffered reader in golang
Asked Answered
S

7

44

I am aware of the specific function in golang from the bufio package.

func (b *Reader) Peek(n int) ([]byte, error)

Peek returns the next n bytes without advancing the reader. The bytes stop being valid at the next read call. If Peek returns fewer than n bytes, it also returns an error explaining why the read is short. The error is ErrBufferFull if n is larger than b's buffer size.

I need to be able to read a specific number of bytes from a Reader that will advance the reader. Basically, identical to the function above, but it advances the reader. Does anybody know how to accomplish this?

Son answered 1/12, 2012 at 15:29 Comment(0)
T
94

Note that the bufio.Read method calls the underlying io.Read at most once, meaning that it can return n < len(p), without reaching EOF. If you want to read exactly len(p) bytes or fail with an error, you can use io.ReadFull like this:

n, err := io.ReadFull(reader, p)

This works even if the reader is buffered.

Tonina answered 28/10, 2013 at 23:25 Comment(5)
This should be the accepted answer. Eliminates the nagging doubt of a "short" read i.e. no need to loop and check for io.EOF's etc. Doc's have a great example here too: golang.org/pkg/io/#ReadFullCopperhead
Maybe it's worth noting that io.ReadFull is just a wrapper for this call: io.ReadAtLeast(reader, p, len(p)) Also in the case of io.ReadFull you should first define the p with its length equals to the size of bytes you want to read, but for io.ReadAtLeast the length of the p can be whatever as long as it's bigger than or equals to the size you want to read.Bobbinet
@Bobbinet is right. The question wording was "specific number of bytes", i.e if I understand the English right, that equals to "exact number of bytes". Besides, quite an actual question if you want to implement message framing over a stream. "Exact", or "specific", is not the same thing as "bigger than or equals". If I need exactly 4 bytes, then what if it will read 400? The question was about this, specifically.Jandel
Actually, io.LimitReader is only useful if you want to limit some code you don't control, e.g. some external library code. If it is your code, then there's no use of it, since it just calls a plain Read() once, which anyway can't read more than you requested. Now the thing, people are asking, and I was trying to find an answer: WHERE exactly it is told the read can't read more than requested? The answer is: NOWHERE. In the Linux docs, the underlying level, they say "usually", and "expected to". Anything in Go just don't say a word. Probably, for guys of Google that was way too obvious?Jandel
I mean, in other places people are suggesting using LimitReader, to limit the maximum number of bytes read. Actually, that's just a redundant safety net, as io.ReadFull() actually does the job, and reads exactly the number you requested. Still, if you look at the source code, you will be infected with doubt, as @Bobbinet pointed out above. ))) What I miss is some tip in Read() docs, saying it actually can't read more than requested.Jandel
R
16
func (b *Reader) Read(p []byte) (n int, err error)

http://golang.org/pkg/bufio/#Reader.Read

The number of bytes read will be limited to len(p)

Redouble answered 1/12, 2012 at 16:44 Comment(2)
This will not 'always' read a specific number of bytes though, it will only limit the bytes read to len(p).Misstate
Also, it may not read at all. According to this solution, you might need to call Read repeatedly until you get the anticipated data. Not all readers are the same. This answer assumes that as if they are.Codd
S
11

TLDR:

my42bytes, err := ioutil.ReadAll(io.LimitReader(myReader, 42))

Full answer:

@monicuta mentioned io.ReadFull which works great. Here I provide another method. It works by chaining ioutil.ReadAll and io.LimitReader together. Let's read the doc first:

$ go doc ioutil.ReadAll
func ReadAll(r io.Reader) ([]byte, error)
     ReadAll reads from r until an error or EOF and returns the data it read. A
     successful call returns err == nil, not err == EOF. Because ReadAll is
     defined to read from src until EOF, it does not treat an EOF from Read as an
     error to be reported. 

$ go doc io.LimitReader
func LimitReader(r Reader, n int64) Reader
     LimitReader returns a Reader that reads from r but stops with EOF after n
     bytes. The underlying implementation is a *LimitedReader.

So if you want to get 42 bytes from myReader, you do this

import (
        "io"
        "io/ioutil"
)

func main() {
        // myReader := ...
        my42bytes, err := ioutil.ReadAll(io.LimitReader(myReader, 42))
        if err != nil {
                panic(err)
        }
        //...
}

Here is the equivalent code with io.ReadFull

$ go doc io.ReadFull
func ReadFull(r Reader, buf []byte) (n int, err error)
    ReadFull reads exactly len(buf) bytes from r into buf. It returns the number
    of bytes copied and an error if fewer bytes were read. The error is EOF only
    if no bytes were read. If an EOF happens after reading some but not all the
    bytes, ReadFull returns ErrUnexpectedEOF. On return, n == len(buf) if and
    only if err == nil. If r returns an error having read at least len(buf)
    bytes, the error is dropped.
import (
        "io"
)

func main() {
        // myReader := ...
        buf := make([]byte, 42)
        _, err := io.ReadFull(myReader, buf)
        if err != nil {
                panic(err)
        }
        //...
}

Compared to io.ReadFull, an advantage is that you don't need to manually make a buf, where len(buf) is the number of bytes you want to read, then pass buf as an argument when you Read

Instead you simply tell io.LimitReader you want at most 42 bytes from myReader, and call ioutil.ReadAll to read them all, returning the result as a slice of bytes. If successful, the returned slice is guaranteed to be of length 42.

Stoichiometric answered 15/5, 2019 at 3:45 Comment(4)
Regarding your last paragraph, the slice will not be guaranteed to be of length 42 bytes. An io.Reader may return fewer bytes than requested. ReadFull will try to read as many bytes as you want, LimitReader will limit the bytes you're going to read. ReadFull simplifies handling the case when you want to really read N bytes.Codd
...or it will return an error (a good thing). + ReadAll already may make a buffer of a bigger size than you need.Codd
This must be an accepted approach. Maybe a little rewording at the end would help, but otherwise it solves the problem and answers the question. I wonder why there's no straightforward function for this. Thanks @navigaidJandel
Actually, io.LimitReader is only useful if you want to limit some code you don't control, e.g. some external library code. If it is your code, then there's no use of it, since it just calls a plain Read() once, which anyway can't read more than you requested. Now the thing, people are asking, and I was trying to find an answer: WHERE exactly it is told the read can't read more than requested? The answer is: NOWHERE. In the Linux docs, the underlying level, they say "usually", and "expected to". Anything in Go just don't say a word. Probably, for guys of Google that was way too obvious? ))))Jandel
T
3

I am prefering Read() especially if you are going to read any type of files and it could be also useful in sending data in chunks, below is an example to show how it is used

fs, err := os.Open("fileName"); 

if err != nil{
    fmt.Println("error reading file")
    return
}

defer fs.Close()

reader := bufio.NewReader(fs)

buf := make([]byte, 1024)

for{
    v, _ := reader.Read(buf) //ReadString and ReadLine() also applicable or alternative

    if v == 0{
        return
    }
    //in case it is a string file, you could check its content here...
    fmt.Print(string(buf))
}
Threecolor answered 2/12, 2013 at 18:12 Comment(0)
W
1

Pass a n-bytes sized buffer to the reader.

Whence answered 1/12, 2012 at 16:24 Comment(1)
This is not guarantee that all n bytes will be fetched by Read(). It could be less without EOF in some marginal cases causing floating error. It's better to allocate buffer of exact size buf := make([]byte, n) and then io.ReadAtLeast(reader, buf, len(buf))Vile
M
0

If you want to read the bytes from an io.Reader and into an io.Writer, then you can use io.CopyN

CopyN copies n bytes (or until an error) from src to dst. It returns the number of bytes copied and the earliest error encountered while copying.

On return, written == n if and only if err == nil.

written, err := io.CopyN(dst, src, n)
if err != nil {
    // We didn't read the desired number of bytes
} else {
   // We can proceed successfully
}
Maggie answered 17/1, 2022 at 22:47 Comment(0)
C
-1

To do this you just need to create a byte slice and read the data into this slice with

n := 512
buff := make([]byte, n)
fs.Read(buff)  // fs is your reader. Can be like this fs, _ := os.Open('file')

func (b *Reader) Read(p []byte) (n int, err error)

Read reads data into p. It returns the number of bytes read into p. The bytes are taken from at most one Read on the underlying Reader, hence n may be less than len(p)

Cuspidate answered 4/7, 2016 at 6:12 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.