How to set the buffer size on a BufferedWriter over a FileWriter

Question

I met a problem with BufferedWriter when I write data to a single file with some threads.

I set the buffer size of the BufferedWriter, but no matter what number I set, it flushes the data to disk when the buffer is 8192 (the default buffer size), not the size I set (here is 16384). Is there a problem with my code?

This is how I'm constructing the BufferedWriter:

new BufferedWriter(new FileWriter(fileName, true), 16384);

This is the full code:

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

public class Test1 {
    public static void main(String[] args) throws IOException {
        for(int i =0;i<10;i++){
            MyThread r = new MyThread();
            Thread t = new Thread(r);
            t.start();
        }
    }
}

class MyThread implements Runnable {
    public void run() {
        String s = "{addffffffkkkljlkj2015dd}\n";
        BufferedWriter bw = null;
        try {
            bw = new BufferedWriter(new FileWriter(
                    "/Users/liaoliuqing/Downloads/1.txt", true),16384);
        } catch (IOException e) {
            e.printStackTrace();
        }
        for(int i =0 ; i<1000; i++){
            try {
                bw.write(String.format("%03d", i)+s);
                //bw.flush();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

Luralurch

I run the program in debian and mac, it get the same result. No matter how I change the buffer size, it has no effect. – Luralurch

And what is the effect you want? – Demakis

if the buffer size set to 16384, it should write to disk when the data is 16384 bytes, not 8192. Now no matter what I set, it write to disk every 8192 bytes. – Luralurch

FileWriter buffers internally: #6108543 The doc says "To specify [character encoding and byte-buffer size] yourself, construct an OutputStreamWriter on a FileOutputStream", but I don't see a constructor for either of those classes that lets you specify a buffer size :( – Hassett

Answer 1

I solve the problem by using OutputStream, not writer, here is the code:

bw = new BufferedOutputStream(
                new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true),165537);

Luralurch

Good for you, but that's weird, since Writer delegates over OutputStream and all the behavior (appending flag, buffer size, exact operations if writing ASCII) is identical. Any other changes? – Mcmorris

Answer 2

Is there a problem with my code?

A few. Mainly: potential IO and concurrency errors. File buffer size might be a lesser concern (and one you can't effectively deal with).

• Trying to open already opened file. All your threads are trying to write into the same file (1.txt). That might be an issue. FileWriter documentation says:

  Some platforms, in particular, allow a file to be opened for writing by only one FileWriter (or other file-writing object) at a time. In such situations the constructors in this class will fail if the file involved is already open.

• Lines might be cut and mixed. If you have several threads with their respective buffers flushing at some point into the same output, you might not even need weird race-conditions or threads stopped right of the middle or a write operation to see your output corrupted.

  As I solution (If your threads must share the same output) you can use a shared object with synchronized access to take care of actual writing. I implemented SafeAppender in my example, but probably there are better alternatives out there.

• No flushing and closing buffers will mean (the tail of) your data will be lost (like tears in the rain). A finally block is usually good to take care of that.

• Also, as stated by other users, BufferedWriter buffer size does not affect the buffer size in FileOutputStream (and so FileWriter). And it looks the java.io and java.nio APIs dont offer any way to mess with that. If you look at the Java library sources you might notice BufferedWriter buffer size just means the amount of chars you store before actually writing into the delegate output. The default size (8192) is optimal for most cases, and increasing it might mean more trouble (potentially losing more data) than benefits.

This is my code, if it serves you:

// https://mcmap.net/q/1201512/-how-to-set-the-buffer-size-on-a-bufferedwriter-over-a-filewriter
public class TestWriter {

public static class SafeAppender {
    private BufferedWriter bw;
    private int users = 0;
    public SafeAppender(File f) throws IOException {
        bw = new BufferedWriter(new FileWriter(f));
    }

    public synchronized void append(String s) throws IOException {
        bw.write(s);
    }
    public synchronized void incrUsers() { 
        users ++; 
    }
    public synchronized void decrUsers() {
        if (--users <= 0) {
            try {
                bw.flush();
                System.err.println("INFO-appender-flush()");
            } catch (Throwable whatever) { /* log-if-you-care*/}
        }
    }
    // Might be called by GC, or not
    @Override protected void finalize() throws Throwable {
        try {
            bw.close();
            System.err.println("INFO-appender-close()");
        } catch (Throwable whatever) { /* log-if-you-care */}
        super.finalize();
    }
}

private static class MyRunnable implements Runnable {
    final static String S = "{addffffffkkkljlkj2015dd}";
    SafeAppender appender;
    String threadId;
    public MyRunnable (SafeAppender a, String tid) {
        appender = a; threadId = tid;
    }

    public void run() {
        appender.incrUsers();
        try {
            for(int i =0 ; i<1000; i++){
                // NOTE: Not a good idea to printStackTrace if each line fails. Let thread fail
                String line = String.format("%s-%03d-%s\n", threadId, i, S);
                appender.append(line);
            }
        } catch (IOException e) {
            System.err.printf("ERROR-%s-%s\n", threadId, e.toString());
        } finally {
            appender.decrUsers();
        }
    }
}

public static void main(String[] args) {
    try {
        File f = File.createTempFile("TestWriter", ".txt");
        System.err.printf("INFO-main-Writing into %s\n", f.getCanonicalPath());
        SafeAppender appender = new SafeAppender (f);
        for(int i =0;i<10;i++){
            MyRunnable r = new MyRunnable(appender, ""+i);
            Thread t = new Thread(r);
            t.start();
        }
    } catch (Throwable e) {
        e.printStackTrace(System.err);
    }
}

}

Mcmorris

Answer 3

FileWriter actually uses its own fixed-size 1024 byte buffer. The BufferedWriter on the other hand, show that it uses and 8192 byte buffer size (default), which can be configured by the user to any other desired size.

And to further muddy the waters, the Java 6 implementation of OutputStreamWriter actually delegates to a StreamEncoder, which uses its own buffer with a default size of 8192 bytes. And the StreamEncoder buffer is user-configurable, although there is no way to access it directly through the enclosing OutputStreamWriter.

Gradatim

This answer appears to have been copied from #6977393, which contains some additional useful links. – Hassett

Indeed it is. But copy would be a wrong word. I could have rephrased it to suggest it i my answer, but the point was to pass the information and no point rewriting it if it's already there. i skipped those links to keep answer minimal and relevant and kept doors open for OP to do some research himself. :) – Gradatim

Answer 4

What you are seeing is not the size of the buffer BufferedWriter, but the size of the buffer used internally by FileWriter. Quoting from the Java Documentation (http://docs.oracle.com/javase/7/docs/api/java/io/FileWriter.html)

The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.

So if you wanted to have a fine grain control on when the data is actually written to the disk you should instantiate your BufferedWriter as

bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File('my_file.txt),true)));

Linguini

How does that help you control the buffer size? – Hassett

I set to this: bw = new BufferedWriter(new OutputStreamWriter( new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true)),65536); bw.write(String.format("%03d", i)+s); bw.flush(); my string is 27340 bytes, so the buffer size is big enough to contain the string. but still it write to disk every 8192 bytes. – Luralurch

My answer was based on the documentation: as it states 'to specify those values' (plural), I assumed that in that case the FileOutputStream would not have a buffer of its own.... – Linguini