why fclose() is not always flushing the data to the disk?

Question

I'm getting some weird results, while trying to write data to files in C.<br/> I thought that fclose() closes the *FILE and flushes the data from its buffer to the file.<br/> But for some reason it only flushes the data in my program sometimes and it doesn't do it other times.

For example: I would run this code, and in my file I can see the two strings. (Perfect, exactly what I want)
But then when I run the code the next 4 times it doesn't change anything in my file. And then when I run it another time, suddenly the 10 extra strings appear (8 from the last times I ran the program and the 2 from now)
(This 4 times is just an example, sometimes it's 5, 8, 10, or even just 2 times before I see the output appear)

I really don't understand this? Shouldn't the data be visible after every time I run the program? Where is this buffer even saved between the different times I run the program, because the program finishes every time, so the memory gets released, right?

(By the way I also tried fflush(fd) before and after fclose(), but that didn't solve the problem)

#include <stdio.h>


int main(int argc, char const *argv[]) {
  FILE * fd;
  fd = fopen("test_file.txt", "a");
  fprintf(fd, "String 1\n");
  fprintf(fd, "String 2\n");
  fclose(fd);
  return 0;
}

Bartholomew

How to you look at the resulting file? What operating system do you use? What filesystem? – Acquittal

Perhaps you leave open the text editor which you use to view the file content, and that does not get updated. You might need to reload the file, and some editors warn you that the file content has changed. – Awake

Test the return value: if (fclose(fd)) perror("fclose");. When there is no error, fclose() includes the equivalent of a fflush(): see C11 7.21.5.1. – Amp

In a case like this you should ideally check the result of all four i/o statements to verify they are supposed to have succeeded. – Awake

What @WeatherVane noted about checking return values from I/O statements is critical. In the posted code, you can't know if ANY of your calls worked. Note also that the data in the fprintf() statements may not actually be written to the file until fclose() is called, and that fclose() can fail after all your other calls appeared to succeed because the data was buffered and not actually written until fclose() flushed it. If your disk gets full or other problems happen, all those previous "successful" fwrite() calls can be wiped out. – Hamburger

An addition to my comment about text editors: although some warn about changes to the file, I have never used a text editor that updates the view automatically, probably because it would be a bad policy. – Awake

"10 extra strings appear" --> What are you using to read the file? – Baobaobab

you need to check the return value of fclose also calling fclose is not ensuring that data is already on the disk. see my answer below. thank you. – Morganstein

When I take your code and run tail -f test_file.txt in another terminal, I get the extra lines every single time. I agree with others the problem is likely in what you use to view the file. – Michaels

Answer 1

Thanks to michael kerrisk we have all the answers we need in linux man pages. Will if you dig in the man pages and see the Notes section (listed below in my answer) you will understand this behavior.

int fclose(FILE *stream); - close a file stream pointed by stream.

DESCRIPTION

The fclose() function flushes the stream pointed to by stream (writing any buffered output data using fflush()) and closes the underlying file descriptor.

NOTES

Note that fclose() flushes only the user-space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, for example, with sync() or fsync().

So its not enough to ensure writing and flushing your buffers, we need to flush the kernel buffers as well. how ? see below: make sure that you are reading my comments in the code as well!

#include <unistd.h>
#include <stdio.h>

int main(int argc, char const *argv[]) {
  FILE * fd;
  fd = fopen("test_file.txt", "a");
  fprintf(fd, "String 1\n");
  fprintf(fd, "String 2\n");
  
  if(fclose(fd) != 0){
/*********************************************************************
fclose failed and errno will be set to endicate the error. be aware 
that we shall 
not call fsync or any other stream manipulation on fd in this case 
because we will get undefined behavior !!!
**********************************************************************/
      ... do work but no more work on this stream (fd)...
  }

  /* ~~ be aware that this is a blocking call ! (see details below) ~~*/
  if(fsync(fd) == -1){
      /*fsync fails*/
      ....
  } 
  return 0;
}

fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) so that all changed information can be retrieved even if the system crashes or is rebooted. This includes writing through or flushing a disk cache if present. The call blocks until the device reports that the transfer has completed

One more important thing, always prefer calling fsync(fd) rather than calling void sync(void); ! why ?

because sync() causes all pending modifications to filesystem metadata and cached file data to be written to the underlying filesystems. usualy we dont want this kind of behavior (and this trigger the kernel to do extra work that is not essential! so dont call it.

this page and answer is not large enough to include all the details and special cases and all error codes! please refer to man pages and to the links belwo:

fclose

fsync

Morganstein

@Bartholomew - i would be glad if you upvote and accept my answer if you find it useful (push the up arrow and V near by the answer start. thanks). – Morganstein

fsync takes a file descriptor, not a FILE pointer. Also, on modern-ish copy-on-write filesystems, fsync ends up flushing all the way to the root, since the metadata updates get caught in the storm. For that reason, it is effectively a no-op on zfs, hsfs, and a good many others. – Recycle