Force write of a file to disk

Question

I'm currently implementing a ping/pong buffering scheme to safely write a file to disk. I'm using C++/Boost on a Linux/CentOS machine. Now I'm facing the problem to force the actual write of the file to disk. Is it possible to do so irrespective of all the caching policies of the filesystem (ext3/ext4) / SO custom rules / RAID controller / harddisk controller ?

Is it best to use plain fread()/fwrite(), c++ ostream or boost filesystem?

I've heard that simply flushing out the file (fflush()) doesn't guarantee the actual write

Oversize

Answer 1

fflush (for FILE*), std::flush (for IOStream) to force your program to send to the OS.

POSIX has

• sync(2) to ask to schedule writing its buffers, but can return before the writing is done (Linux is waiting that the data is send to the hardware before returning).

• fsync(2) which is guaranteed to wait for the data to be send to the hardware, but needs a file descriptor (you can get one from a FILE* with fileno(3), I know of no standard way to get one from an IOStream).

• O_SYNC as a flag to open(2).

In all cases, the hardware may have it's own buffers (but if it has control on it, a good implementation will try to flush them also and ISTR that some disks are using capacitors so that they are able to flush whatever happens to the power) and network file systems have their own caveat.

Penis

And for C++ streams there is the manipulator std::flush instead of fflush. – Superordinate

thank you! so I have to force my program to commit to filesystem first (fflush/flush), then to force SO to commit to disk controller (sync). Can you show me some proof-of-concept code? – Oversize

@JoachimPileborg, that's what I intended to signify by the too terse (f)flush, I've clarified. – Penis

I feel obliged to point out that none of these guarantee that the data is physically written to disk. It may very well be present only in the HDD cache or other mystical places inside your computer. – Potato

@rubenvb: that's exactly my concern. Unfortunately disabling the write-cache on the RAID controller is not an option due to the great loss in performance. Moreover the solution should be software only. – Oversize

@G_G then you're out of luck. The solution will definitely be OS-dependent, and I believe most OSes don't have guarantees even for the relevant low-level system calls. – Potato

sync causes the system to schedule the writes to disk. It doesn't block until they have occurred. For transactional integrity, you need to open the file with O_SYNC, or use fsync (which requires knowing the file descriptor). – Ambush

@JamesKanze, oops, I committed to memory a guaranteed present on Linux. – Penis

NOTE: the file is a small snapshot (less than 1Kb) of some important data the system needs to preserve in case of a complete power failure. The system uses it to restore it's status after such power down. The data are sampled and committed to disk every seconds so it is not essential that each write is performed, but at least it is executed every X seconds with a, let's say, 90% confidence. – Oversize

@Penis I don't see where it's guaranteed even in Linux. According to the man page for Linux, the current behavior is for it to wait (since version 1.3.20), but that is listed in the section "Bugs", which suggests that it might change. (Having to wait for all outstanding writes to finish is likely to make it very slow.) – Ambush

Re your edit: Posix guarantees transactional integrity for fsync and if the O_SYNC flag is used. But in practice, that's only theory. The system should take into account any hardware buffering, in so far as possible---and well designed hardware will make it possible, but not all hardware is well designed. As for remote file systems... You're right to warn. – Ambush

Answer 2

You can use fsync()/fdatasync() to force(Note 1) the data onto the storage. Those requres a file descriptor, as given by e.g. open(). The linux manpage have more linux specific info, particularly on the difference of fsync and fdatasync.

If you don't use file desciptors directly, many abstractions will contain internal buffers residing in your process.

e.g. if you use a FILE*, you first have to flush the data out of your application.

//... open and write data to a FILE *myfile
fflush(myfile);
fsync(fileno(myfile));

• Note 1: These calls force the OS to ensure that any data in any OS cache is written to the drive, and the drive acknowledges that fact. Many hard-drives lie to the OS about this, and might stuff the data in cache memory on the drive.

Enculturation

@G_G When you're done with the FILE*, to close the file, then use fclose() – Enculturation

sorry, I mean when using directy the file descriptors; do we need an explicit close(fd) call? – Oversize

@G_G If you use FILE*, use fclose. If you use filedescriptors use close(). If you're pulling the file descriptor out of a FILE*, you still have a FILE*, so use fclose (which will also close the file descriptor ) – Enculturation

Answer 3

Not in standard C++. You'll have to use some sort of system specific IO, like open with the O_SYNC flag under Unix, and then write.

Note that this is partially implicit by the fact that ostream (and in C, FILE*) are buffered. If you don't know exactly when something is written to disk, then it doesn't make much sense to insist on the transactional integrity of the write. (It wouldn't be too hard to design a streambuf which only writes when you do an explicit flush, however.)

EDIT:

As a simple example:

class SynchronizedStreambuf : public std::streambuf
{
    int myFd;
    std::vector<char> myBuffer;

protected:
    virtual int overflow( int ch );
    virtual int sync();

public:
    SynchronizedStreambuf( std::string const& filename );
    ~SynchronizedStreambuf();
};

int SynchronizedStreambuf::overflow( int ch )
{
    if ( myFd == -1 ) {
        return traits_type::eof();
    } else if ( ch == traits_type::eof() ) {
        return sync() == -1 ? traits_type::eof() : 0;
    } else {
        myBuffer.push_back( ch );
        size_t nextPos = myBuffer.size();
        myBuffer.resize( 1000 );
        setp( &myBuffer[0] + nextPos, &myBuffer[0] + myBuffer.size() );
        return ch;
    }
}

int SynchronizedStreambuf::sync()
{
    size_t toWrite = pptr() - &myBuffer[0];
    int result = (toWrite == 0 || write( myFd, &myBuffer[0], toWrite ) == toWrite ? 0 : -1);
    if ( result == -1 ) {
        close( myFd );
        setp( NULL, NULL );
        myFd = -1;
    } else {
        setp( &myBuffer[0], &myBuffer[0] + myBuffer.size() );
    }
    return result;
}

SynchronizedStreambuf::SynchronizedStreambuf( std::string const& filename )
    : myFd( open( filename.c_str(), O_WRONLY | O_CREAT | O_SYNC, 0664 ) )
{
}

SynchronizedStreambuf::~SynchronizedStreambuf()
{
    sync();
    close( myFd );
}

(This has only been superficially tested, but the basic idea is there.)

Ambush