BufferedWriter not writing everything to its output file

Question

I have a Java program that reads some text from a file, line by line, and writes new text to an output file. But not all the text I write to my BufferedWriter appears in the output file after the program has finished. Why is that?

The details: the program takes a CSV text document and converts it into SQL commands to insert the data into a table. The text file has more than 10000 lines which look similar to following:

2007,10,9,1,1,1006134,19423882

The program seems to work fine except it just stops in the file randomly half way through creating a new SQL statement having printed it into the SQL file. It looks something like:

insert into nyccrash values (2007, 1, 2, 1, 4, 1033092, 259916);
insert into nyccrash values (2007, 1, 1, 1, 1, 1020246, 197687);
insert into nyccrash values (2007, 10, 9, 1

This happens after about 10000 lines but several hundred lines before the end of the file. Where the break happens is between a 1 and a ,. However, the characters doesn't seem important because if I change the 1 to a 42 the last thing written to the new file is 4, which is cutting off the 2 from that integer. So it seems like the reader or writer must just be dying after writing/reading a certain amount.

My Java code is as follows:

import java.io.*;

public class InsertCrashData
{
    public static void main (String args[])
    {
        try
        {   
            //Open the input file.
            FileReader istream = new FileReader("nyccrash.txt");
            BufferedReader in = new BufferedReader(istream);
            //Open the output file.
            FileWriter ostream = new FileWriter("nyccrash.sql");
            BufferedWriter out = new BufferedWriter(ostream);
            String line, sqlstr;

            sqlstr = "CREATE TABLE nyccrash (crash_year integer, accident_type integer, collision_type integer, weather_condition integer, light_condition integer, x_coordinate integer, y_coordinate integer);\n\n"; 
            out.write(sqlstr);

            while((line = in.readLine())!= null)
            {
                String[] esa = line.split(",");
                sqlstr = "insert into nyccrash values ("+esa[0]+", "+esa[1]+", "+esa[2]+", "+esa[3]+", "+esa[4]+", "+esa[5]+", "+esa[6]+");\n";
                out.write(sqlstr);
            }
        }
        catch(Exception e)
        {
            System.out.println(e);
        }
    }
}

Abvolt

Answer 1

You need to close your OutputStream which will flush the remainder of your data:

out.close();

The default buffer size for BufferedWriter is 8192 characters, large enough to easily hold hundreds of lines of unwritten data.

Meave

@Abvolt - because the last couple of hundred lines are still in the buffer ... and won't be written until the writer is closed or flushed. It is a buffered writer ... – Kevyn

In fact, the FileWriter has to perform charset encoding under the hood, which is a buffered operation on its own. Without flush or close, there can be up to 8192 pending bytes in the default setup, even without BufferedWriter. Since we have the year 2022 now, it’s important to point out that the preferred idiom is the try-with-resources statement rather than calling close() manually. – Anticholinergic

Answer 2

You must close() your BufferedWriter. You must close() your BufferedWriter because it IS-A Writer and thus implements AutoCloseable, which means (emphasis added) it is

A resource that must be closed when it is no longer needed.

Some people say you must first call flush() for your BufferedWriter before calling close(). They are wrong. The documentation for BufferedWriter.close() notes that it "Closes the stream, flushing it first" (emphasis added).

The documented semantics of flushing (flush()) are

Flushes this stream by writing any buffered output to the underlying stream

So, you must close, and close will flush any buffered output.

Your output file does not include all the text you wrote to your BufferedWriter because it stored some of that text in a buffer. The BufferedWriter never emptied that buffer, passing it through to the file, because you never told it to do so.

---

Since Java 7, the best way to ensure an AutoCloseable resource, such as a BufferedWriter, is closed when it is not longer need is to use automatic resource management (ARM), also known as try-with-resources:

 try (BufferedWriter out = new BufferedWriter(new FileWriter(file))) {
    // writes to out here
 } catch (IOException ex) {
    // handle ex
 }

You must also close your BufferedReader when it is no longer need, so you should have nested try-with-resources blocks:

 try (BufferedReader in = new BufferedReader(new FileReader("nyccrash.txt")) {
    try (BufferedWriter out = new BufferedWriter(new FileWriter("nyccrash.sql"))) {
       // your reading and writing code here
    }
 } catch (IOException ex) {
    // handle ex
 }

Do not be tempted (as other answers here suggest) just to call close() at the end of your method, when your code has "finished" using the writer. That will not work if your writing code throws an exception, and in particular if it throws an IOException.

Kremer

You don’t need to nest try statements; a try statement supports more than one resource. try(BufferedReader in = …; BufferedWriter out = …) { … } catch(IOException ex) { … } – Anticholinergic

Answer 3

A resource that must be closed when it is no longer needed.

finally {
    out.close();//this would resolve the issue
    }

Some things to consider:

• BufferedWriter.close() flushes the buffer to the underlying stream, so if you forget to flush() and don't close, your file may not have all the text you wrote to it.
• BufferedWriter.close() also closes the wrapped Writer. When that's a FileWriter, this will ultimately close a FileOutputStream and tell the OS that you're done writing to the file.
• The garbage collector will automatically call close(), not on the BufferedWriter or the wrapped FileWriter, but on the FileOuputStream. So the OS will be happy, but you have to wait for the GC.
• However, you always want to release OS resources as soon as you no longer need them. This goes for open files, database connections, print queues ... anything. Trust me on this one.
• BufferedWriter.close() does clear up the internal character buffer, so that memory will be available for garbage collection, even while the BufferedWriter itself remains in scope.

So, Always close your resources (not just files) when you're done with them.

If you really want a peek under the covers, most of the Java API's source is available. BufferedWriter is here.

Impudent

Actually GC will close the FileDescriptor, not the FileOutputStream. – Humdrum

Answer 4

Your code does not appear to be closing the writer after you've finished writing to it. Add an out.close() (preferably in a finally block) and it should work properly.

Holland

I cannot understand why the output is not written to the file prior to closing the bufferedWritter. Can you please be kind enough to teach me why that is happening – Primal

@KasunSiyambalapitiya if you were to close() the BufferedWriter then the remaining output would be flushed to the file, the problem in this question is that the BufferedWriter never gets closed at all, so some of the output is still left in the internal buffer and doesn't get written to the underlying file stream. – Holland

Answer 5

you dint close your BufferedWriter.close it inside a finally block

   finally {
    out.close();//this would resolve the issue
    }

Whitted

I cannot understand why the output is not written to the file prior to closing the bufferedWritter. Can you please be kind enough to teach me why that is happening – Primal

Because of buffering. It's a BufferedWriter, remember? – Humdrum

Answer 6

Always close your resources (not just files) when you're done with them.

 finally {
    out.close();//this would resolve the issue
    }

There might be situations when you want to flush the buffer without closing the file. In these situations you can use the flush-method.

Digress

Answer 7

Since you're using BufferedWriter you can also flush the buffer when appropriate:

out.flush()

This will write the rest of the buffer to the actual file. Close-method also flushes the buffer and closes the file.

out.close()

There might be situations when you want to flush the buffer without closing the file. In these situations you can use the flush-method.

You can also use BuffredWriter's newline-method instead of adding \n to the end of the line. Newline-method uses system specific line separator so your code works on different platforms.

out.newLine()

Cecilia

Answer 8

According to documentation it is no use calling flush() method. If you intent to use FileWriter then flush() would help you out.

Basically in this case, you just need to close, BufferedWriter.close() only. This will flush the remainder of your data.

create finally block and put the close method inside so that it will put all data without missing.

 finally {
    out.close();
    }

Solley

Nowhere in any documentation is it stated that 'it is no use calling flush()'. – Humdrum