read() of big 6GB file fails on x86_64

Question

Here is the description of my problem:

I want to read a big file, about 6.3GB, all to memory using the read system call in C, but an error occurs. Here is the code:

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <limits.h>

int main(int argc, char* argv[]) {
    int _fd = open(argv[1], O_RDONLY, (mode_t) 0400);
    if (_fd == -1)
        return 1;
    off_t size = lseek(_fd, 0, SEEK_END);
    printf("total size: %lld\n", size);
    lseek(_fd, 0, SEEK_SET);
    char *buffer = malloc(size);
    assert(buffer);
    off_t total = 0;
    ssize_t ret = read(_fd, buffer, size);
    if (ret != size) {
        printf("read fail, %lld, reason:%s\n", ret, strerror(errno));
        printf("int max: %d\n", INT_MAX);
    }
}

And compile it with:

gcc read_test.c

then run with:

./a.out bigfile

output:

total size: 6685526352
read fail, 2147479552, reason:Success
int max: 2147483647

The system environment is

 3.10.0_1-0-0-8 #1 SMP Thu Oct 29 13:04:32 CST 2015 x86_64 x86_64 x86_64 GNU/Linux

There two places I don't understand:

1. Reading fails on a big file, but not on a small file.
2. Even if there is an error, it seems that the errno is not correctly set.

Floreneflorentia

well the call successfully read 2147479552 bytes. you need to loop till you consumed all data.btw? how much ram do you have in total? – Awad

Why? There are few situations where you really need an entire file in memory. – Phillane

is your system limiting the available memory? have you tried ulimit -s unlimited ? – Furunculosis

Related: https://mcmap.net/q/823965/-read-from-files-blocking-vs-non-blocking-behavior/694576 – Fong

If you're using POSIX functions such as open() and read(), you can also use POSIX stat() and/or fstat() to get the size of a file directly. – Puca

Depending on what you're going to do with this huge file once you have it in memory, mmap may be more appropriate than read. – Brannon

Answer 1

The read system call can return a smaller number than the requested size for multiple reasons, a positive non zero return value is not an error, errno is not set in this case, its value is indeterminate. You should keep reading in a loop until read returns 0 for end of file or -1 for an error. It is a very common bug to rely on read to read a complete block in a single call, even from regular files. Use fread for simpler semantics.

You print the value of INT_MAX, which is irrelevant to your issue. The size of off_t and size_t are the interesting ones. On your platform, 64 bit GNU/Linux, you are lucky that both off_t and size_t are 64 bit long. ssize_t has the same size as size_t by definition. On other 64 bit platforms, off_t might be smaller than size_t, preventing correct assessment of the file size, or size_t might be smaller than off_t, letting malloc allocate a block smaller than the file size. Note that in this case, read will be passed the same smaller size because size would be silently truncated in both calls.

Heterolysis

Thanks a lot! When I keep reading in a loop until 0 or -1 reached, everything works fine. – Floreneflorentia

@zhanglistar: there is no definite answer to this question: for small chunks, fread may be faster because of the bufferization performed by default in the standard I/O package; for large chunks, it depends on the actual implementation. Note that fread is the portable solution. read() is a system call standardized in Posix, it is not available on all systems. – Heterolysis

Answer 2

You should only bail on the read if it returns -1. From man page:

On success, the number of bytes read is returned (zero indicates end of file), and the file position is advanced by this number. It is not an error if this number is smaller than the number of bytes requested;

My guess is that at a 2G boundary on your file system, a read() can read a short buffer.

Bourgogne

Answer 3

Try #define _FILE_OFFSET_BITS 64 for open, and #define _LARGEFILE64_SOURCE for lseek64. then you can read write file large than 2GB

Hypogynous

Answer 4

read() system call will not be able to read huge data in one-short. That depends on many factors like internal buffer of kernel, device driver implementation of the media. In your example you are trying to check if the read() has read the data of length size, if not then print failure. You need to keep reading the data until the read bytes is 0, also you need to check the return code returned by read(), if it is -1, then it means that there is some failure in reading in which case you need to check errno being set.

Also I would suggest not to allocate huge memory in one shot, even if the system if capable of allocating huge memory as it's not a good implementation. Consider breaking the size to some chucks if possible.

Rhynd