Allocate file on disk without zeroing

Question

I need to allocate huge file without zeroing it's content. I'm producing this steps fopen => ftruncate => fclose => mmap => (...work...) => munmap with huge file sizes (hundreds of gigabytes). App hangs on termination for a few minutes while system is trying zeroing file bytes – IMHO because of ftruncate usage.

ftruncate(ofd, 0);

#ifdef HAVE_FALLOCATE

    int ret = fallocate(ofd, 0, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(HAVE_POSIX_FALLOCATE)

    int ret = posix_fallocate(ofd, 0, cache_size);
    if (ret == -1) {
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }

#elif defined(__APPLE__)

    fstore_t store = {F_ALLOCATECONTIG, F_PEOFPOSMODE, 0, cache_size, 0};
    int ret = fcntl(ofd, F_PREALLOCATE, &store);
    if (ret == -1) {
        store.fst_flags = F_ALLOCATEALL;
        ret = fcntl(ofd, F_PREALLOCATE, &store);
    }
    if (ret == -1) { // read fcntl docs - must test against -1
        printf("Failed to expand file to size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
        exit(-1);
    }
    struct stat sb;
    ret = fstat(ofd, &sb);
    if (ret != 0) {
        printf("Failed to write to file to establish the size.\n");
        exit(-1);
    }
    //ftruncate(ofd, cache_size); <-- [1]

#endif

It seems it does not work with commented line [1]. But uncommenting this line produces file zeroing which I am trying to avoid. I really don't care dirty file content before writing. I just wanna avoid hang on app termination.

SOLUTION:

According to @torfo's answer, replaced all my Apple-related code with this few lines:

unsigned long long result_size = cache_size;
int ret = fcntl(ofd, F_SETSIZE, &result_size);
if(ret == -1) {
    printf("Failed set size %llu (errno %d - %s).\n", cache_size, errno, strerror(errno));
    exit(-1);
}

But only works for superuser!

Darnley

deleted (ftruncate() should create a sparse file, too, so it just seems your OS / filesystem doesn't support sparse files) – Betsybetta

If you need to make a huge file without zeroing its content, just ftruncate without doing the other things you do. That will create a sparse file if your filesystem supports it. fallocate and the other calls will do the opposite, they do preallocate and zero the disk blocks. You run the risk of running out of disk space when writing to a sparse file, but you don't waste time zeroing it. – Simsar

@Simsar app with ftruncate hangs for a few minutes on termination via Ctrl+C. – Darnley

Note allocating previously used disk space without zeroing it and without "sparsing it" creates a potential security hole. – Periodontal

Answer 1

This is MacOS X, apparently.

You can try to replace the ftruncate call with

fcntl(ofd, F_SETSIZE, &size);

(note needs root privilege and might create a security hole because it might provide access to old file contents that was previously there, so must be handled with extreme caution. The "dirty file content" you don't care about might actually be the user's bank account passwords he deleted a week ago...)

MacOS X doesn't really support sparse files - It does create and maintain them, but its file system driver is very eager to fill the holes as soon as possible.

Periodontal

Thanks! fcntl(ofd, F_SETSIZE, &result_size); works like a charm! – Darnley

Yep, I don't care security problems for this use case. It's a plots generation for HDD mining: github.com/r-majere/mjminer/pull/3 – Darnley