Concurrent writing to a file

Question

In go, how can I control the concurrent writing to a text file?

I ask this because I will have multiple goroutines writing to a text file using the same file handler.

I wrote this bit of code to try and see what happens but I'm not sure if I did it "right":

package main

import (
    "os"
    "sync"
    "fmt"
    "time"
    "math/rand"
    "math"
)


func WriteToFile( i int, f *os.File, w *sync.WaitGroup ){
    //sleep for either 200 or 201 milliseconds
    randSleep := int( math.Floor( 200 + ( 2 * rand.Float64() ) ) )
    fmt.Printf( "Thread %d waiting %d\n", i, randSleep )
    time.Sleep( time.Duration(randSleep) * time.Millisecond )

    //write to the file
    fmt.Fprintf( f, "Printing out: %d\n", i )
    //write to stdout
    fmt.Printf( "Printing out: %d\n", i )
    w.Done()
}

func main() {
    rand.Seed( time.Now().UnixNano() )

    d, err := os.Getwd()
    if err != nil {
        fmt.Println( err )
    }
    filename := d + "/log.txt"

    f, err := os.OpenFile( filename, os.O_CREATE | os.O_WRONLY | os.O_TRUNC, 0666 )

    if err != nil {
        fmt.Println( err )
    }
    var w *sync.WaitGroup = new(sync.WaitGroup)
    w.Add( 10 )

    //start 10 writers to the file
    for i:=1; i <= 10; i++ {
        go WriteToFile( i, f, w )
    }

    //wait for writers to finish
    w.Wait()

}

I half expected that the output would show something like this in the file instead of the coherent output I got:

Printing Printing out: 2
out: 5
Poriuntitng: 6

Essentially, I expected the characters to come out incoherently and interweaved due to a lack of synchronization. Did I not write code that would coax this behavior out? Or is some mechanism during calls to fmt.Fprintf synchronizing the writing?

Vancevancleave

I'm not sure how it works on other operating systems .. but generally a lock is acquired while writing to StdOut on Windows at least at line breaks. – Bowker

@SimonWhitehead So you're saying that the operating system might be handling the lock at a lower level and that it might not be directly handled in go. – Vancevancleave

Answer 1

There are many ways to control concurrent access. The easiest is to use a Mutex:

var mu sync.Mutex

func WriteToFile( i int, f *os.File, w *sync.WaitGroup ){
    mu.Lock()
    defer mu.Unlock()
    // etc...
}

As to why you're not seeing problems, Go uses operating system calls to implement file access, and those system calls are thread safe (emphasis added):

According to POSIX.1-2008/SUSv4 Section XSI 2.9.7 ("Thread Interactions with Regular File Operations"):

All of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2008 when they operate on regular files or symbolic links: ...

Among the APIs subsequently listed are write() and writev(2). And among the effects that should be atomic across threads (and processes) are updates of the file offset. However, on Linux before version 3.14, this was not the case: if two processes that share an open file description (see open(2)) perform a write() (or writev(2)) at the same time, then the I/O operations were not atomic with respect updating the file offset, with the result that the blocks of data output by the two processes might (incorrectly) overlap. This problem was fixed in Linux 3.14.

I would still use a lock though, since Go code is not automatically thread safe. (two goroutines modifying the same variable will result in strange behavior)

Suchlike

Answer 2

A simple approach to controlling concurrent access is via a service goroutine, receiving messages from a channel. This goroutine would have sole access to the file. Access would therefore be sequential, without any race problems.

Channels do a good job of interleaving requests. The clients write to the channel instead of directly to the file. Messages on the channel are automatically interleaved for you.

The benefit of this approach over simply using a Mutex is that you start viewing your program as a collection of microservices. This is the CSP way and leads to easy composition of large systems from smaller components.

Upanchor

Communicating Sequential Processes (en.wikipedia.org/wiki/Communicating_sequential_processes) - a process algebra from Tony Hoare at Oxford University – Upanchor

Inside the service goroutine, would you open/close the file every time a new message comes in? or would you leave it open for as long as the goroutine lives? – Griseldagriseldis

That depends on the application, but the 'normal' case would involve opening the file once at the start, then ranging over the messages on the input channel, writing each one to the file. When the channel has closed then the file can be closed. – Upanchor