Transactionally writing files in Node.js

Question

I have a Node.js application that stores some configuration data in a file. If you change some settings, the configuration file is written to disk.

At the moment, I am using a simple fs.writeFile.

Now my question is: What happens when Node.js crashes while the file is being written? Is there the chance to have a corrupt file on disk? Or does Node.js guarantee that the file is written in an atomic way, so that either the old or the new version is valid?

If not, how could I implement such a guarantee? Are there any modules for this?

Stipple

Answer 1

What happens when Node.js crashes while the file is being written? Is there the chance to have a corrupt file on disk? Or does Node.js guarantee that the file is written in an atomic way, so that either the old or the new version is valid?

Node implements only a (thin) async wrapper over system calls, thus it does not provide any guarantees about atomicity of writes. In fact, fs.writeAll repeatedly calls fs.write until all data is written. You are right that when Node.js crashes, you may end up with a corrupted file.

If not, how could I implement such a guarantee? Are there any modules for this?

The simplest solution I can come up with is the one used e.g. for FTP uploads:

1. Save the content to a temporary file with a different name.
2. When the content is written on disk, rename temporary file to destination file.

The man page says that rename guarantees to leave an instance of newpath in place (on Unix systems like Linux or OSX).

Evolutionary

I can confirm that in OSX this does not solve the problem altogether, however it does reduce the incidence of file corruption – Swore

Answer 2

write-file-atomic will do what you need. It writes to temporary file, then rename. That's safe.

Child

Answer 3

fs.writeFile, just like all the other methods in the fs module are implemented as simple wrappers around standard POSIX functions (as stated in the docs).

Digging a bit in nodejs' code, one can see that the fs.js, where all the wrappers are defined, uses fs.c for all its file system calls. More specifically, the write method is used to write the contents of the buffer. It turns out that the POSIX specification for write explicitly says that:

Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of IEEE Std 1003.1-2001 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.

So it seems it is pretty safe to write, as long as the size of the buffer is smaller than PIPE_BUF. This is a constant that is system-dependent though, so you might need to check it somewhere else.

Armoured

According to manpages.courier-mta.org/htmlman7/pipe.7.html, PIPE_BUF is required to be at least 512 bytes, and in Linux is 4.096 bytes. Thanks for pointing this out, as this is really good and interesting stuff to know :-) – Stipple