How to create a 10MB file filled with "000000..." data in golang?

Question

I am intended to use fdatasync in a system like log or diskqueue. The first thing is to create a 10MB file with "000000..." in file system like ext4. But I don't know how to do it properly.

Atrocious

Answer 1

jnml@fsc-r630:~/src/tmp/SO/16797380$ ls -l
celkem 4
-rw-rw-r-- 1 jnml jnml 186 kvě 29 07:54 main.go
jnml@fsc-r630:~/src/tmp/SO/16797380$ cat main.go 
package main

import (
        "log"
        "os"
)

func main() {
        f, err := os.Create("foo.bar")
        if err != nil {
                log.Fatal(err)
        }

        if err := f.Truncate(1e7); err != nil {
                log.Fatal(err)
        }
}
jnml@fsc-r630:~/src/tmp/SO/16797380$ go run main.go 
jnml@fsc-r630:~/src/tmp/SO/16797380$ ls -l
celkem 4
-rw-rw-r-- 1 jnml jnml 10000000 kvě 29 07:55 foo.bar
-rw-rw-r-- 1 jnml jnml      186 kvě 29 07:54 main.go
jnml@fsc-r630:~/src/tmp/SO/16797380$ 

Katherine

I actually prefer this method to the one I provided as I think the use of Truncate is quite clean and elegant. – Freon

I love this one. It is elegant. But it's still a sparse file. I don't sure this is the right answer. – Atrocious

Answer 2

If you are using unix, then you can create a sparse file very quickly. A sparse file is filled with zero (ascii NUL) and doesn't actually take up the disk space until it is written to, but reads correctly.

package main

import (
    "log"
    "os"
)

func main() {
    size := int64(10 * 1024 * 1024)
    fd, err := os.Create("output")
    if err != nil {
        log.Fatal("Failed to create output")
    }
    _, err = fd.Seek(size-1, 0)
    if err != nil {
        log.Fatal("Failed to seek")
    }
    _, err = fd.Write([]byte{0})
    if err != nil {
        log.Fatal("Write failed")
    }
    err = fd.Close()
    if err != nil {
        log.Fatal("Failed to close file")
    }
}

Which produces a 10MB file called output.

$ ls -l output 
-rw-r--r-- 1 user user 10485760 May 28 18:58 output
$ du -hs output 
4.0K    output

Update much later

I solved exactly this problem in rclone. Namely preallocating files without writing the data, or creating a sparse file. You can't do it directly from the standard library and it isn't cross platform, but using the fallocate syscall with the FALLOC_FL_KEEP_SIZE flag is the way to go in linux. You can also do this in windows.

Here are links to the relevant code in windows and linux and here is the linux code:

var (
    fallocFlags = [...]uint32{
        unix.FALLOC_FL_KEEP_SIZE,                             // Default
        unix.FALLOC_FL_KEEP_SIZE | unix.FALLOC_FL_PUNCH_HOLE, // for ZFS #3066
    }
    fallocFlagsIndex int32
)

// preAllocate the file for performance reasons
func preAllocate(size int64, out *os.File) error {
    if size <= 0 {
        return nil
    }
    index := atomic.LoadInt32(&fallocFlagsIndex)
again:
    if index >= int32(len(fallocFlags)) {
        return nil // Fallocate is disabled
    }
    flags := fallocFlags[index]
    err := unix.Fallocate(int(out.Fd()), flags, 0, size)
    if err == unix.ENOTSUP {
        // Try the next flags combination
        index++
        atomic.StoreInt32(&fallocFlagsIndex, index)
        fs.Debugf(nil, "preAllocate: got error on fallocate, trying combination %d/%d: %v", index, len(fallocFlags), err)
        goto again

    }
    // FIXME could be doing something here
    // if err == unix.ENOSPC {
    //  log.Printf("No space")
    // }
    return err
}

Whitver

A sparse file doesn't satisfy my requirement. I wanna use fdatasync rather than fsync, so the metadata of the file should not be changed in later wirte-operations. – Atrocious

From my reading of the man page I think fdatasync will work fine on sparse files. None of the metadata needs to be updated when you write to a sparse file. – Whitver

Would you please give me some url-links? I still got confusing. – Atrocious

Relevant man pages fdatasync and stat. Writing to the sparse sections of the file could change blkcnt_t so maybe it will update the metadata. Anyway, if you are after ultimate performance then don't write a sparse file as there is more overhead when writing to it. If you want to create a 1 GB backing file instantly then do use a sparse file! – Whitver

That's exactly what I worry about. But now, I have to give up this sync-thing, just let os auto-flush by itself. In mostly situation, it seems safty enough. And it also have good peformance compare to sync. – Atrocious

Answer 3

Try this:

package foo

import "io"

func WriteBytes(w io.Writer, c byte, n uint) {
    buf := make([]byte,0x1000)

    for i := 0 ; i < len(buf) ; i++ {
        buf[i] = c
    }

    for i := 0 ; i < n >> 24 ; i++ {
        w.Write(buf)
    }

    w.Write(buf[:n & 0xfff])
}

Grisly

This could make a sparse file too. I'm not sure about it. So if I must choose an answer, I like the Truncate best. – Atrocious

@Atrocious This is very unlikely to create a sparse file. Ideally, you'd use the syscall.Fallocate() function for this on Linux. – Grisly

Answer 4

This will create a 10000000 byte file in the current directory named testfile.

package main

import (
    "fmt"
    "os"
)

func main() {
    data := make([]byte, int(1e7), int(1e7)) // Initialize an empty byte slice
    f, err := os.Create("testfile")
    if err != nil {
        fmt.Printf("Error: %s", err)
    }
    defer f.Close()
    size, err := f.Write(data) // Write it to the file
    if err != nil {
        fmt.Printf("Error: %s", err)
    }
    fmt.Printf("%d bytes written", size)
}

Hope that helps.

Freon