Best way to Pipe InputStream to OutputStream [duplicate]

Question

I was to trying to find the best way to pipe the InputStream to OutputStream. I don't have an option to use any other libraries like Apache IO. Here is the snippet and output.

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.channels.FileChannel;

public class Pipe {
    public static void main(String[] args) throws Exception {

        for(PipeTestCase testCase : testCases) {
            System.out.println(testCase.getApproach());
            InputStream is = new FileInputStream("D:\\in\\lft_.txt");
            OutputStream os = new FileOutputStream("D:\\in\\out.txt");

            long start = System.currentTimeMillis();            
            testCase.pipe(is, os);
            long end = System.currentTimeMillis();

            System.out.println("Execution Time = " + (end - start) + " millis");
            System.out.println("============================================");

            is.close();
            os.close();
        }

    }

    private static PipeTestCase[] testCases = {

        new PipeTestCase("Fixed Buffer Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[1024];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                }
            }
        },

        new PipeTestCase("dynamic Buffer Read") {           
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                byte[] buffer = new byte[is.available()];
                while(is.read(buffer) > -1) {
                    os.write(buffer);   
                    buffer = new byte[is.available() + 1];
                }
            }
        },

        new PipeTestCase("Byte Read") {         
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                int c; 
                while((c = is.read()) > -1) {
                    os.write(c);    
                }
            }
        }, 

        new PipeTestCase("NIO Read") {          
            @Override
            public void pipe(InputStream is, OutputStream os) throws IOException {
                FileChannel source      = ((FileInputStream) is).getChannel(); 
                FileChannel destnation  = ((FileOutputStream) os).getChannel();
                destnation.transferFrom(source, 0, source.size());
            }
        }, 

    };
}


abstract class PipeTestCase {
    private String approach; 
    public PipeTestCase( final String approach) {
        this.approach = approach;           
    }

    public String getApproach() {
        return approach;
    }

    public abstract void pipe(InputStream is, OutputStream os) throws IOException;
}

Output (~4MB input file) :

Fixed Buffer Read
Execution Time = 71 millis
============================================
dynamic Buffer Read
Execution Time = 167 millis
============================================
Byte Read
Execution Time = 29124 millis
============================================
NIO Read
Execution Time = 125 millis
============================================

'Dynamic Buffer Read' uses available() method. But it is not reliable as per java docs

It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

'Byte Read' seems to be very slow.

So 'Fixed Buffer Read' is the best option for pipe? Any thoughts?

Enfeoff

did you mean the apache IO is better – Hassock

Answer 1

I would say a fixed buffer size is the best/easiest to understand. However there are a few problems.

• You're writing the entire buffer to the output stream each time. For the final block the read may have read < 1024 bytes so you need to take this into account when doing the write (basically only write number of bytes returned by read()

• In the dynamic buffer case you use available(). This is not a terribly reliable API call. I'm not sure in this case inside a loop whether it will be ok, but I wouldn't be suprised if it was implemented sub-optimally in some implementations of InputStream.

• The last case you are casting to FileInputStream. If you intend for this to be general purpose then you can't use this approach.

Probably

Agree with all three points. The fixed buffer case needs to fixed. The dynamic buffer MHO is just broken - I wouldn't be surprised if a situation occurs where available() returns 0 before the end is reached (i.e. when reading from network connection). – Piton

@Mike Q, NIO is not option for me right now. the interface defines the InputStream and OutputStream. – Enfeoff

I normally use fixed buffer size at 16k or more, as most people have a spare 16k of ram these days :) Also, if possible it's good to know how many bytes are coming in so you don’t end the writing too soon. – Wesson

Answer 2

Java 9

Since Java 9 one can use this method from InputStream:

public long transferTo(OutputStream out) throws IOException

Pre Java 9

A one-liner from apache commons:

IOUtils.copy(inputStream, outputStream);

Documentation here. There are multiple copy methods with different parameters. It is also possible to specify the buffer size.

Egghead

This doesn't answer the question. The question said the answer can't use other libraries. – Wasp

you're right. he didn't answer the question. But, the answer did give me a solution since I could use 3rd party. I would edit the answer to acknowledge the fact it doesn't answer his need, but it's a useful answer for others like me. So my thanks. – Davison

Source library is written explicitely at the start. Google indexes titles best, hence this question has a pretty high view count. The "other libraries" remark is not as highly indexed. – Egghead

It may not answer @OP's needs, but it answers mine, and I can't very well ask a new question can I? It would just be linked to this question and then I'd be told to not ask questions that has already been answered. Since search engines send us here, there's even more of a reason to have these extra answers: They help those who aren't OP, but have almost the same problem. – Aerophagia

Answer 3

I came across this, and the final read can cause problems.

SUGGESTED CHANGE:

public void pipe(InputStream is, OutputStream os) throws IOException {
  int n;
  byte[] buffer = new byte[1024];
  while((n = is.read(buffer)) > -1) {
    os.write(buffer, 0, n);   // Don't allow any extra bytes to creep in, final write
  }
 os.close ();

I also agree that 16384 is probably a better fixed buffer size than 1024.

IMHO...

Mathematician

Buffer size should be a parameter then, different use cases call for different buffer sizes. – Lazes

@AurélienOoms No they don't. The code in this answer will work for any buffer size greater than zero. – Cooe

Answer 4

java.io contains PipedInputStream and PipedOutputStream

PipedInputStream input = new PipedInputStream();
PipedOutputStream output = new PipedOutputStream (input);

write to input and it will be visible in output as an Outputstream. Things can work the other way around as well

Loch

The OP already has both input and output - how do you connect both ends with these piped streams? – Knob

the op question is not really about pipes... it's more about how to copy one file to another. A pipe is more about passing an output stream as the input stream for another process. – Loch

"write to input" - did you mean write to output? – Blacklist