ext4/fsync situation unclear in Android (Java)

Question

Tim Bray's article "Saving Data Safely" left me with open questions. Today, it's over a month old and I haven't seen any follow-up on it, so I decided to address the topic here.

One point of the article is that FileDescriptor.sync() should be called to be on the safe side when using FileOutputStream. At first, I was very irritated, because I never have seen any Java code doing a sync during the 12 years I do Java. Especially since coping with files is a pretty basic thing. Also, the standard JavaDoc of FileOutputStream never hinted at syncing (Java 1.0 - 6). After some research, I figured ext4 may actually be the first mainstream file system requiring syncing. (Are there other file systems where explicit syncing is advised?)

I appreciate some general thoughts on the matter, but I also have some specific questions:

1. When will Android do the sync to the file system? This could be periodic and additionally based on life cycle events (e.g. an app's process goes to the background).
2. Does FileDescriptor.sync() take care of syncing the meta data? That is syncing the directory of the changed file. Compare to FileChannel.force().
3. Usually, one does not directly write into the FileOutputStream. Here's my solution (do you agree?):

   FileOutputStream fileOut = ctx.openFileOutput(file, Context.MODE_PRIVATE);
   BufferedOutputStream out = new BufferedOutputStream(fileOut);
   try {
       out.write(something);
       out.flush();
       fileOut.getFD().sync();
   } finally {
       out.close();
   }
   

Marybellemarybeth

Answer 1

Android will do the sync when it needs to -- such as when the screen turns off, shutting down the device, etc. If you are just looking at "normal" operation, explicit sync by applications is never needed.

The problem comes when the user pulls the battery out of their device (or does a hard reset of the kernel), and you want to ensure you don't lose any data.

So the first thing to realize: the issue is when power is suddenly lost, so a clean shutdown can not happen, and the question of what is going to happen in persistent storage at that point.

If you are just writing a single independent new file, it doesn't really matter what you do. The user could have pulled the battery while you were in the middle of writing, right before you started writing, etc. If you don't sync, it just means there is some longer time from when you are done writing during which pulling the battery will lose the data.

The big concern here is when you want to update a file. In that case, when you next read the file you want to have either the previous contents, or the new contents. You don't want to get something half-way written, or lose the data.

This is often done by writing the data in a new file, and then switching to that from the old file. Prior to ext4 you knew that, once you had finished writing a file, further operations on other files would not go on disk until the ones on that file, so you could safely delete the previous file or otherwise do operations that depend on your new file being fully written.

However now if you write the new file, then delete the old one, and the battery is pulled, when you next boot you may see that the old file is deleted and new file created but the contents of the new file is not complete. By doing the sync, you ensure that the new file is completely written at that point so can do further changes (such as deleting the old file) that depend on that state.

Zucker

I would have expected the flush() to make sure everything is written to disc - essentially by calling sync(). Isn't that the case? – Cymograph

@a_horse_with_no_name: No, it's not. flush and sync are two different operations: flush just flushes intermediate buffers; sync actually writes to storage. See e.g. #2341110 – Caneghem

Answer 2

fileOut.getFD().sync(); should be on the finally clause, before the close().

sync() is way more important than close() considering durability.

So, everytime you want to 'finish' working on a file you should sync() it before close()ing it.

posix does not guarantee that pending writes will be written to disk when you issue a close().

Chazan