atomic file creation on Linux?

Question

I need to create a file if it does not exist, in a way that another process trying to create this file would fail. I need the file be considered "created" even before the creating process finished writing the actual data to it.

I read about O_EXCL flag to open(), so it seems that the solution exists, I have a few questions however:

1. do you have experience with this technique? How good is it? (I guess I can't have a DB-level atomicity, but but good enough is... well, enough)
2. should I immediately close the file after open() so that it is considered created, and then reopen it for writing?
3. are there any subtleties to be aware of?

Positronium

Answer 1

The open() man page says your method may fail on NFS.

From the section on O_EXCL:

When used with O_CREAT, if the file already exists it is an error and the open() will fail. In this context, a symbolic link exists, regardless of where it points to. O_EXCL is broken on NFS file systems; programs which rely on it for performing locking tasks will contain a race condition.

And it suggests a more general solution:

The solution for performing atomic file locking using a lockfile is to create a unique file on the same file system (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful.

See the "Using Files as Locks" section of this Web page for more details on the various issues and approaches.

Anarch

yes, i've seen this, so my questions were for the case I am not working with NFS. Also, the alternative solution is obscure for me - how creating a unique file per process helps 2 processes see that they are trying to work with the same file? If you understand this, care to explain? – Positronium

link and unlink calls on NFS are not idempotent or synchronized. Use mkdir instead. – Waffle

@davka: this method is using a link as a lock. Whatever file the link currently points to, "owns" the lock. (And each file is unique to a process, so you you have process locking). – Anarch

@Positronium – fcntl is supposed to work over NFS, so as an alternative you could use that to synchronise your processes on the write instead of on the open. – Schlemiel

Answer 2

POSIX says:

If O_CREAT and O_EXCL are set, open() shall fail if the file exists. The check for the existence of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_EXCL and O_CREAT set.

So other processes using O_EXCL will consider it opened as soon as it is created.

Schlemiel

thanks, this is more unambiguous. I wandered whether the file is considered "created" right after open(). This seem to say that yes – Positronium

@Positronium – I guess it says "yes, it should be", but I don't see any point in having the flag at all without this guarantee. BSD manpages say "This may be used to implement a simple exclusive access locking mechanism." Without the guarantee that wouldn't work. – Schlemiel