Flush disk write cache

Question

When the policy for a disk in Windows XP and Vista is set to enable write caching on the hard disk, is there a way to flush a file that has just been written, and ensure that it has been committed to disk?

I want to do this programmatically in C++.

Closing the file does perform a flush at the application level, but not at the operating system level. If the power is removed from the PC after closing the file, but before the operating system has flushed the disk write cache, the file is lost, even though it was closed.

Wearisome

Do you mean programatically? The question does not make that clear – Glyconeogenesis

We assume that your question is about programming, but you haven't specified the development environment. – Boomer

Answer 1

You haven't specified the development environment, so:

.Net

IO streams have a .Flush method that does what you want.

Win32 API

There is the FlushFileBuffers call, which takes a file handle as argument.

EDIT (based on a comment from the OA): FlushFileBuffers does not need administrative privileges; it does only if the handle passed to it is the handle for a volume, not for a single file.

Boomer

It appears that this would work. Unfortunately, the application must be able to run without administrative rights. – Wearisome

I couldn't find this .NET method. As far as I can see, the only place the framework uses FlushFileBuffers is in SerialStream. I suspect that p/invoking FlushFileBuffers is safe bet. – Agro

I found that calling FlushFileBuffers via pInvoke can cause an exception (https://mcmap.net/q/585831/-flushing-log-to-disk-exception-in-verifyoshandleposition/4540). Under .NET 4, it's easier and safer to just call FileStream.Flush(true) as @Rai suggests (https://mcmap.net/q/563769/-how-to-ensure-all-data-has-been-physically-written-to-disk). – Wolfy

Answer 2

.NET FileStream.Flush() will NOT flush the Windows cache for that file content; Flush() only flushes the .NET internal file buffer. In .NET 4.0, Microsoft fixed the problem by adding an optional parameter to Flush() which if set true causes FlushFileSystemBuffers to be called. In .NET 3.5 and below your only choice is to call FlushFileBuffers via pinvoke. See MSDN'sFileStream.Flush community comment for how to do this.

Rai

WARNING: .Net 4.0 Flush(true) doesn't fix it!!!! MS bug report here says file.Flush(true) is broken, then fixed, but doesn't say what version or service pack it was fixed in! Sounds like bug was if internal .NET FileStream buffer is empty, the Flush(true) did nothing?? – Rai

Answer 3

You should not fix this at the time you close the file. Windows will cache, unless you open the file passing FILE_FLAG_WRITE_THROUGH to CreateFile().

You may also want to pass FILE_FLAG_NO_BUFFERING; this tells Windows not to keep a copy of the bytes in cache.

This is more efficient than FlushFileBuffers(), according to the CreateFile documentation on MSDN.

See also file buffering and file caching on MSDN.

Mullins

Answer 4

You should also note, that your data might not get flushed to the actual disk, even when invoking a flush method of your frameworks API.

Calling the flush method will only tell the kernel to flush its pages to disk. However, if you have the disk write-cache turned on, it is allowed to delay the actual writing process indefinitely.

In order to ensure that your data gets written to the physical layer you have to turn of the write cache in your operating system. This most often comes with a performance penalty up to one or two orders of magnitude when dealing with a lot of small io-operations. Battery based support (UPS) or disks that accept commands to flush the disk write-cache are another option to deal with this problem.

Integrate

Windows will force disk cache flushes for important data unless the disk policy has been set to "Turn off Windows write-cache buffer flushing" – Bise

Of course it will. However, If the harddisk's write-cache is turned on the disk can stall the write-operation. Read, for example: support.microsoft.com/kb/259716/EN-US Quote: "By enabling write caching, file system corruption and/or data loss could occur if the machine experiences a power, device or system failure and cannot be shutdown properly." – Integrate

Actually, Windows will force flush caching on FlushFileBuffers() no matter if write cache buffering was turned on, and even if write cache flushing (earlier known as "advanced performance checkbox") was turned on. – Saltwort

Answer 5

From the microsoft documents you would use _flushall and link in COMMODE.OBJ to ensure that all buffers were committed to disk.

Ciera

We also had to add the "c" mode option during fopen() – Reactivate

Answer 6

See here: https://jeffpar.github.io/kbarchive/kb/066/Q66052/

When you initially open your file using fopen, include the "c" mode option as the LAST OPTION:

fopen( path, "wc") // w - write mode, c - allow immediate commit to disk

Then when you want to force a flush to disk, call

_flushall()

We made this call before calling

fclose()

We experienced the exact issue you described and this approach fixed it.

Note that this approach does NOT required Administrative rights, which FlushFileBuffers does require, as others have mentioned.

From that above site:

"Microsoft C/C++ version 7.0 introduces the "c" mode option for the fopen() function. When an application opens a file and specifies the "c" mode, the run-time library writes the contents of the file buffer to disk when the application calls the fflush() or _flushall() function. "

Reactivate

Answer 7

You can open/create the file with the FileOptions.WriteThrough flag, which will cause the file to write directly into the disk, bypassing any caches.

E.g.

var file = File.Open(
    "1.txt",
    new FileStreamOptions
    {
        Options = FileOptions.WriteThrough
    });

// - OR -
var file = new FileStream(
    "1.txt", 
    FileMode.Create, 
    FileAccess.Write, 
    FileShare.None, 
    4096, 
    FileOptions.WriteThrough)

Canned

Answer 8

Have you tried using Sysinternals Sync utility to flush all cached file data?

This will flush all devices including removable media.

.\sync.exe -r

Hang