how standard specify atomic write to regular file(not pipe or fifo)?

Question

The posix standard specified that when write less than PIPE_BUF bytes to pipe or FIFO are granted atomic, that is, our write doesn't mix with other processes'. But I failed to find out how standard specify about regular file. I mean it's true that when we write less than PIPE_BUF, it will also granted be atomic. But I want to know does regular file have such limitation? I mean, the pipe has the capacity, so that when write to the pipe and beyond its capacity, kernel will put the writer to sleep, so other process will get chance to write, but regular file seems that doesn't have to have such limitation, am i right?

What I'm doing is several processes generate log to a file. Of course, with O_APPEND set.

Mannuela

Pipes are different from files in that pipes are entirely sequential - unable to seek. Each process that has a file open, on the other hand, has it's own separate pointer to where the next write is going to take place, so I'm not sure the comparison you're trying to make is really useful. However, pread() and pwrite() might be at least partially what you're looking for... – Enlarger

No..I was writing a program that have several processes output to log file. I need to be sure that the out put doesn't mix up. Also I don't want to use flock or semaphore liked methods, cause they are not very effective. – Mannuela

pl. check once writev. The documentation says its write operation is atomic. linux.die.net/man/2/writev "The data transfers performed by readv() and writev() are atomic: the data written by writev() is written as a single block that is not intermingled with output from writes in other processes.. – Stable

@Tanmoy I see, If I want to be sure the write is atomic, even though I have only one buffer, I can use writev() to achieve this. right? But how about write()? – Mannuela

Related: Are Unix reads and writes to a single file atomically serialized? – Charismatic

Answer 1

Quote from http://pubs.opengroup.org/onlinepubs/9699919799/toc.htm (Single UNIX Specification, Version 4, 2010 Edition):

This volume of POSIX.1-2008 does not specify behavior of concurrent writes to a file from multiple processes. Applications should use some form of concurrency control.

The specification does address how semantics of writes regarding writes occur in case of multiple readers, but as you can see from above, the behaviour for multiple, concurrent writers is not defined by the specification.

Note above talks about files. For pipes and FIFOs the PIPE_MAX semantics apply, that concurrent writes are guaranteed to be non-divisible up to PIPE_MAX bytes.

Write requests to a pipe or FIFO shall be handled in the same way as a regular file with the following exceptions:

Write requests of {PIPE_BUF} bytes or less shall not be interleaved with data from other processes doing writes on the same pipe. Writes of greater than {PIPE_BUF} bytes may have data interleaved, on arbitrary boundaries, with writes by other processes, whether or not the O_NONBLOCK flag of the file status flags is set.

For real file systems the situation is complex. Some local file systems may enforce atomic writes up to arbitrary sizes (memory limit) by locking a file handle during writing, some might not (I tried to look at ext4 logic, but lost track somewhere around http://lxr.linux.no/linux+v3.5.3/fs/jbd2/transaction.c#L147).

For non-local file systems the result is more or less for grabs. Just don't try concurrent writing on a networked file system without some form of explicit locking (or you're positively absolutely sure about the semantics of the network file system you're using).

BTW, O_APPEND guarantees that all writes by different processes go to the end of the file. However as SUS above notes, if the writes are really concurrent (occuring at the same time), then the behavior is undefined. On earlier uniprocess and non-pre-emptive UNIXes this didn't really matter, as a call to write(2) completed before someone else got a chance to write...

This question could be answered definitely for specific combination of operating system (Linux?) and file system (ext4?). A general answer? As SUS reads -- "undefined behavior".

Merrifield

Answer 2

I think this is useful to you: "the data written by writev() is written as a single block that is not intermingled with output from writes in other processes", so you can use writev

Dross

Answer 3

Several writers to a file may mix up things. But files opened with O_APPEND are appended atomically per write access.

If you want to keep to the C stdio interface instead of the lower level one, fopene the file with "a" or "a+" (which map to O_APPEND), set up a buffer large enough that there is no need to write inside your records and use fsync to force the write when you are done building them. I'm not sure it is guaranteed by POSIX (C says nothing about that).

Horsley

I just want to know, if write to regular file is always atomic, now matter how many bytes I write, or have some limitation like PIPE_BUF. Even though file doesn't have capacity limitation like pipe. But current linux kernel is preemptive and multi-core system makes thing more complicate. – Mannuela

Answer 4

There is the ultimate solut8ion to all questions of atomicity; a mutex. Wrap your writes to the log file in a mutex and all will be done atomically.

A simpler solution might be to use the GLOG libraries from Google. A fantastic logging system, far better than anything I ever dreamed up, free, not-GPL, and atomic.

Periotic

Answer 5

One way to interleave them safely would be to have all writers lock the file, write, and unlock.

Functions that can be used for locking are flock(), lockf(), and fcntl().

Beware that ALL writers must lock (and they should all use the same mechanism to do the locking) or one that doesn't bother getting a lock could still write at the same time as another that holds a lock.

Gawk