How to write large binary data fast in Java? [duplicate]

Question

I am writing an STL file which consists of an 80 byte header, a 4 byte integer, and then records of 50 bytes each consisting of floats and a short integer.

Using a RandomAccessFile I can easily write the data, but it's horribly slow. This uses the same interface as DataOutputStream. If there is an easy way to buffer the Data Output Stream, I could use that, but the annoying part is the need to write out all the records, and at the end count the number of triangles output and write that integer to bytes 81-84.

The straightforward, but slow way, just focusing on the majority of the work, which is writing each facet:

public static void writeBinarySTL(Triangle t, RandomAccessFile d) throws IOException {
    d.writeFloat((float)t.normal.x);
    d.writeFloat((float)t.normal.y);
    d.writeFloat((float)t.normal.z);
    d.writeFloat((float)t.p1.x);
    d.writeFloat((float)t.p1.y);
    d.writeFloat((float)t.p1.z);
    d.writeFloat((float)t.p2.x);
    d.writeFloat((float)t.p2.y);
    d.writeFloat((float)t.p2.z);
    d.writeFloat((float)t.p3.x);
    d.writeFloat((float)t.p3.y);
    d.writeFloat((float)t.p3.z);
    d.writeShort(0);
}

Is there any elegant way to write this kind of binary data into a blockwise, fast I/O class?

It also occurs to me that the STL file format is supposed to be low byte first, and Java is probably high byte first. So perhaps all my writeFloats are in vain, and I will have to find a manual conversion so that it comes out in little-endian form?

If I have to, I would be willing to close the file, reopen at the end in a randomaccessfile, seek to byte 81 and write the count.

So this edit is in response to the two answers to the question that should work. The first is adding a BufferedWriter. The result is amazingly fast. I know this laptop is high end with an SSD, but I wasn't expecting this kind of performance, let alone from Java. Just by buffering output, 96Mb file written in 0.5 seconds, and 196Mb in 1.5 seconds.

To see whether nio would offer even higher performance, I attempted to implement the solution by @sturcotte06. The code does not attempt to write the header, I just focused on the 50 byte records for each triangle.

public static void writeBinarySTL2(Shape3d s, String filename) {
    java.nio.file.Path filePath = Paths.get(filename);

    // Open a channel in write mode on your file.
    try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE)) {
        // Allocate a new buffer.
        ByteBuffer buf = ByteBuffer.allocate(50 * 1024);
        ArrayList<Triangle> triangles = s.triangles;
        // Write your triangle data to the buffer.
        for (int i = 0; i < triangles.size(); i += 1024) {
            for (int j = i; j < i + 1024; ++j)
                writeBinarySTL(triangles.get(j), buf);
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer's data.
        }
        channel.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

I tried both WRITE (which the documentation says requires an existing file) and CREATE which the documentation claims will either write to an existing file or make a new one.

Both options fail to create the file Sphere902_bin2.stl

java.nio.file.NoSuchFileException: Sphere902_bin2.stl
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at 

sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at edu.stevens.scad.STL.writeBinarySTL2(STL.java:105)

I don't believe the code that writes the bytes is relevant, but this is what I came up with:

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.z)));
    buf.putShort((short)0);
}

Here is an MWE showing the code not working when writing beyond the size of the buffer:

package language;
import java.io.*;
import java.nio.*;
import java.nio.file.*;
import java.nio.channels.*;

public class FastWritenio {
    public static void writeUsingPrintWriter() throws IOException {
        PrintWriter pw = new PrintWriter(new FileWriter("test.txt"));
        pw.print("testing");
        pw.close();
    }
    public static void writeUsingnio(int numTrials, int bufferSize, int putsPer) throws IOException {
        String filename = "d:/test.dat";
        java.nio.file.Path filePath = Paths.get(filename);
        WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.READ);
        ByteBuffer buf = ByteBuffer.allocate(bufferSize);
        for (int t = 0; t < numTrials; ++t) {
            for (int i = 0; i < putsPer; i ++) {
                buf.putInt(i);
            }
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer's data.
            // Without this line, it crashes:  buf.flip();
            // but with it this code is very slow.
        }
        channel.close();
    }
    public static void main(String[] args) throws IOException {
        writeUsingPrintWriter();
        long t0 = System.nanoTime();
        writeUsingnio(1024*256, 8*1024, 2048);
        System.out.println((System.nanoTime()-t0)*1e-9);

    }
}

Adherence

If there is an easy way to buffer the Data Output Stream, you mean, like a BufferedOutputStream? new DataOutputStream(new BufferedOutputStream(new FileOutputStream("out.dat"))) – Gessner

Well,can you buffer a RandomAccessFile, or do I have to write out the file,as a DataOutputStream, close, then reopen, seek to the count and write? – Adherence

You can of course do that. Or you could instead count the triangles, then write the header, then write the count, then write the triangles. – Gessner

In all answers, the speed improvement comes from reducing the number of calls to the operating system. A context switch from user space (where you app runs) to kernel space is very expensive. RandomAccessFile (and FileOutputStream as well) calls the operating system for every method call. BufferedOutputStream first accumulates a configurable number of bytes (8kb by default) before calling the OS. For a 4-byte write, that's 2048x fewer context switches. – Cony

That isn't quite true. If you write an odd number of bytes, you will be writing to the same block twice. So for example with a disk block size of 4k, and a buffer size of 4k-1, you would be writing each block twice. All things being equal, a bigger buffer is better up to a point, but then if you make a gigantic buffer, you won't even write to disk until later, so with multitrhreading and cache issues, there are some other things to think about. – Adherence

Why this question is closed? Linked question is different, because it about text files, not binary. And how to reopen question? PS: I pressed on flag and make rquest – Ingenue

Answer 1

Use nio's ByteBuffer class:

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putFloat((float)t.normal.x);
    // ...
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    writeBinarySTL(t, buf);

    // Flip from write mode to read mode.
    buf.flip();

    // Write your buffer's data.
    channel.write(buf);
}

ByteBuffer tutorial

ByteBuffer javadoc

WriteableByteChannel javadoc

Files javadoc

EDIT:

public static boolean tryWriteBinarySTL(Triangle t, ByteBuffer buf) {
    final int triangleBytes = 50; // set this.
    if (buf.remaining() < triangleBytes) {
       return false;
    }

    buf.putFloat((float)t.normal.x);
    // ...
    return true;
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    for (Triangle triangle : triangles) {
        while (!tryWriteBinarySTL(triangle, buf) ) {
            // Flush buffer.
            buf.flip();
            while (buf.hasRemaining()) {
                channel.write(buf);
            }

            buf.flip();
        }
    }

    // Write remaining.
    buf.flip();
    channel.write(buf);
}

Brush

The documentation of ByteBuffer is incredibly unclear. I can use putInt() to write integers, and putShort() to write a short, but if writing values that do not evenly fit into buffer size, do I continue to write to the buffer (ie does it autoflush? When do I have to flip? Why do I have to flip? I just want to flush basically... – Adherence

Buffers aren't flushable; you flush memory towards the disks. A buffer is in memory. The buffer will throw if full and cannot put your data structure in it. Buffers in itself does nothing to the file; it needs to be written to a file channel. Follow my example. – Brush

Flip is used to switch between read and write mode for the buffer. Basically a data source will write to the buffer, then flip it for the data destination to read it. When you flip, the buffer stores the current size of the buffer and limits it; it will therefore contain the correct number of bytes. – Brush

A disk is going to efficiently write multiples of block size. How does this code handle it when I write 50 bytes at a time, which will never be a multiple of block size? – Adherence

How is this relevent to the question? The OS handles disks; this is not your responsibility. You're trying to micro-optimize. What you need to optimize right now is the number of system calls. Fill your buffer with data, be it headers or payload. Call write with your filled buffer. If you have less data than one buffer, then it will never be efficient, but it does not need to be, as you have minimal data. There's buffering at the OS level and at the disk level, don't try to be smarter than today's systems; you'll end up with worse performance and more complicated code. – Brush

I am adding code to the question that does not work, trying to implement your answer. You are dead wrong on the performance issue of blocks, as someone who has written this in C++, but that's not relevant to the question – Adherence

It has nothing to do with disks. The OS will align data with the underlying block size regardless of your buffer size. The performance issue you experienced is the misalignment of your app buffer size and the OS buffer size, which incurs a memcpy penalty. C++ has nothing to do with that; java, c, c++, or whatever language you use does not let you bypass the OS (unless you're writing your own OS, which is out of the scope of this question). – Brush

I will give you the answer if you will correct your code to show how to write after flipping. Should it be buf.flip(); channel.write(); This fails. Do I buf.flip() again afterwards? – Adherence

The example is complete and self contained; your problem is not algorithmic, but​ environmental. You cannot write because of access rights or something like that. Trim your code to writing a single byte to a temp file and fix forward from there. – Brush

No, the write works fine until the buffer size is exceeded. – Adherence

IO is written in a loop. You read one buffer from source, then write that buffer to destination. Flipping the buffer switches the buffer from filling it to consuming it. Basically, you create a buffer (defaults to fill mode). You fill it at source. You flip it to consume mode. Then you write the buffer to destination, until the buffer is consumed. You can then flip it back to fill mode and so on until source is exhausted and destination has all data. Since you know the number of triangle you have, I don't know why you'd need more than on loop iteration; hence why it is not in the answer. – Brush

You need to flip twice per loop iteration, since each iteration will fill data and consume data. – Brush

So the second buf.flip() is necessary, but that makes the code extremely slow. – Adherence

Well of course, you're calling write syscall at every triangle. Basically, you coupled your business logic (triangles) to IO details. You need decouple those two in order to write efficient IO. To be efficient, your buffer must be aligned with the OS buffer (8k on linux and most OS) and writes must be called with full buffers only (except last buffer, which might be semi empty). – Brush

I don't know what you are talking about, but the MWE writes integers (no triangles) and fills a buffer with 2k int (8k bytes) and then calls buf.flip() and writes. See the bottom of the question where I added a simpler MWE – Adherence

You also missed the incorrect indentation. flip is being called only after 1024 triangles. I fixed the indentation to make it clearer – Adherence

Let us continue this discussion in chat. – Brush

@Brush "You need to flip twice per loop iteration" is not correct. You must flip once and either compact or clear once. – Solemn

Answer 2

If there is an easy way to buffer

There is:

DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(...));

Solemn

This seems really great. I am going to add this to my question. I want to understand the nio solution, but with this one I got an insane rate of 196Mb/1.5 seconds – Adherence

Answer 3

DataOutputStream is not a particularly speedy solution. Create a plain old FileOutputStream, put a BufferedOutputStream around it, and then write your own code to write your data as bytes. The Float and Double classes have helper functions for this. doubleToLongBits, for example.

If that's not fast enough for you, then read up on NIO2.

Oliana

It is at least as speedy as your own code, and it already works. – Solemn

I disagree having read the source. But I upvoted the buffer answer. – Oliana

As far as using streams I don’t see a particular problem in DataOutputStream. It does not use multi byte writes, but given a underlying BufferrdOutputSzream that would be optimized anyway. If you want you can implement writeInt and writeLong, this is used by float and double as well. I am not sure if it would help anything since the data would need to be copied twice. I don’t think you can implement it faster. – Slumber

@Oliana I've read the source too and it is much as I would write it. If you have a specific suggestion you should provide it. – Solemn