How to ensure all data has been physically written to disk?

Question

I understand that .NET FileStream's Flush method only writes the current buffer to disk, but dependent on Windows' disk driver and the hard disk firmware this is no guarantee that the data is actually physically written to disk.

Is there a .NET or Win32 method that can give me this guarantee? So if there is power loss one nanosecond after the call to this method comes back, I can still be sure that everything is OK?

Sophey

Answer 1

Under Windows, look at FlushFileBuffers (Win32 API).

Desired

Thank you :) I have quickly done a performance test and FileStream.Flush() is way too fast to be true. FlushFileBuffers on FileStream's SafeFileHandle is as slow as I would expect it (100 times slower than Flush() in my test) – Sophey

I found that calling FlushFileBuffers can cause an exception (https://mcmap.net/q/585831/-flushing-log-to-disk-exception-in-verifyoshandleposition/4540). Under .NET 4, it's easier and safer to just call FileStream.Flush(true) as @Sasha suggests (https://mcmap.net/q/563769/-how-to-ensure-all-data-has-been-physically-written-to-disk). – Shatzer

The file data that's buffered in the file system cache to be written to disk. That data is normally lazily written, based on the position of the disk write head. Having a gigabyte of cached data is technically possible so it can take quite a while. If this is important to you then consider the FileOptions.WriteThrough option instead. – Scruffy

Is there a way to apply this logic to Registry as well? – Textualist

Answer 2

Stefan S. said:

I understand that .NET FileStream's Flush method only writes the current buffer to disk

No, .NET FileStream's Flush only writes the .NET buffers to the OS cache, it does not flush the OS cache to disk. Sadly the MSDN doc on this class doesn't say that. For .NET < 4.0, you'll have to call Flush + Win32's FlushFilebuffers:

using System.Runtime.InteropServices;
. . .

// start of class:
[DllImport("kernel32", SetLastError=true)]
private static extern bool FlushFileBuffers(IntPtr handle);
. . .

stream.Flush();     // Flush .NET buffers to OS file cache.
#pragma warning disable 618,612 // disable stream.Handle deprecation warning.
if (!FlushFileBuffers(stream.Handle))   // Flush OS file cache to disk.
#pragma warning restore 618,612
{
  Int32 err = Marshal.GetLastWin32Error();
  throw new Win32Exception(err, "Win32 FlushFileBuffers returned error for " + stream.Name);
}

For .NET 4.0, you can instead use the new flush(true) method. 11/09/2012 update: MS bug report here says it's broken, then fixed, but doesn't say what version or service pack it was fixed in! Sounds like bug was if internal .NET FileStream buffer is empty, the Flush(true) did nothing??

Sasha

Answer 3

Well, you could close the file... that would probably do it. In reality, with HAL abstraction, virtualization, and disk hardware now having more processing power and cache memory than computers did a few years ago, you're going to have to live with hoping the disk does its job.

The transactional file system never really materialized ;-p Of course, you could perhaps look at using a database as a back end, and use the transaction system of that?

Aside: note that not all streams even guarantee to Flush() - for example, GZipStream etc retain a working buffer of uncommitted data even after a flush - the only way to get it to flush everything is to Close() it.

Unfold

Technically, with writing to a database, theres no guarantee that a power outage or other catastrophic failure won't lose the write or corrupt the database somehow, although its far more likely to survive than a simple filesystem write. – Yvor

yes, but you can wrap the "mark this as done" and "here's the results" in the same transaction – Unfold

@Yvor If a database system doesn't guarantee that then it's either broken (at least I consider a DBMS that's not ACID broken) or running on a broken system (OS, hardware, whatever). – Spindling

Answer 4

I've noticed that the .NET 4 #Flush(true) doesn't actually write to the disk. We were having strange issues with corrupted data and I found this bug report on the MS site:

The details tab for the bug report has a test program you can run that will show the issue;

1. Write a bunch of data to disk
2. fs.Flush(true). This takes no time (much faster than can possibly written to the disk).
3. Use the win32 API FlushFileBuffers. This takes a long time.

I'm changing over to the win32 FlushFileBuffers call...

Sandstorm

fs.Flush(true) is working for me just fine. Windows 10 x64 creators update, .NET 4.5 – Gober

@jjxtra: Did you test with a network server and physically pulling the plug? – Scholz

@Scholz nope, only tested with local file system, so can't speak for that scenario – Gober

@jjxtra: It will appear to work unless you cause the disks to fail by unplugging them. You need the network server to know you triggered the race. – Scholz

Answer 5

The file data that's buffered in the file system cache to be written to disk. That data is normally lazily written, based on the position of the disk write head. Having a gigabyte of cached data is technically possible so it can take quite a while. If this is important to you then consider the FileOptions.WriteThrough option instead.

Scruffy

This has been proven to be ineffective as well. The OS will still cache. – Crumley

@trevster344, Do you have a source for that? – Proven

@Proven FILE_FLAG_WRITE_THROUGH used to be broken with SATA drives: disruptivesql.wordpress.com/2012/05/08/sata-and-write-through I don't know the current state of affairs, but I wouldn't trust it without further research and testing. – Spindling

Answer 6

There is a simple answer to flushing the content of the buffer to disk. After your WriteAllText function, open file, close it, and reset it

here is an example

My.Computer.FileSystem.WriteAllText(yourfilename, "hello", False, System.Text.Encoding.ASCII)
FileOpen(1, yourfilename, OpenMode.Input)
FileClose(1)
Reset()

Rustproof

Answer 7

There's simply too many levels of abstraction to be absolutely sure that the data is written to the disc, right down to the hardware level.

Not brilliantly performant or foolproof, but how about re-opening the file once it is written in a seperate process and checking the size or contents?

Disentangle

That would be pointless, because it doesn't guarantee that the data has been physically written to disk. – Spindling

Not just non-foolproof, but useless. The re-open request goes through the caching layers, same as anything else. – Secretin