BufferedInputStream into byte[] to be send over a Socket to a Database

Question

I have been looking around for an answer to this, but couldn't really find anything on it. Earlier today, I asked how I could make a File into a String through a byte array, and then back again, for retrieval later.

What people told me, was that I had to just store the byte array, to avoid nasty encoding issues. So now I've started working on that, but I have now hit a wall.

Basically, I used unbuffered streams before, to turn a file into a byte array. This works good in theory, but it takes up a lot of memory which eventually will cast the heap size exception. I should use buffered streams instead (or so I am told), and the problem I have now, is going from a BufferedInputStream to a byte[]. I've tried to copy and use the methods found in this documentation

http://docs.guava-libraries.googlecode.com/git/javadoc/index.html?com/google/common/io/package-summary.html

Where I exchange unbuffered streams for buffered streams. The only issue, is that I can't directly turn a buffered output stream into a byte array, as I can with an unbuffered stream.

Help? :)

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public final class BufferedByteStream {

    private static final int BUF_SIZE = 1024000;

    public static long copy(BufferedInputStream from, BufferedOutputStream to) throws IOException { 
        byte[] buf = new byte[BUF_SIZE];
        long total = 0;
        while(true) {
            int r = from.read(buf);
            if(r == -1) {
                break;
            }
            to.write(buf, 0, r);
            total += r;
        }
        return total;
    }

    public static byte[] toByteArray(BufferedInputStream in) throws IOException {
        BufferedOutputStream out = new BufferedOutputStream(new ByteArrayOutputStream());
        copy(in, out);
        return out. // <--- Problem is here
    }
}

EDIT:

I am still getting Heap Space errors. So I will now post all the code:

main.java

import java.io.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import jserver.io.BufferedByteStream;
/**
*
* @author Vipar
*/
public class main {
    public static void main(String[] args) {
    File f = new File("<doesn't matter>");
        try {
            byte[] buf;
            try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f))) {
                buf = BufferedByteStream.toByteArray(bis);
                bis.close();
            }
            File f2 = new File("<doesn't matter>");
            try (FileOutputStream fos = new FileOutputStream(f2)) {
                fos.write(buf);
                fos.close();
            }
        } catch (FileNotFoundException ex) {
            Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(main.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
}

BufferedByteStream.java

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

public final class BufferedByteStream {

    private static final int BUF_SIZE = 1024000;

    public static long copy(BufferedInputStream from, BufferedOutputStream to) throws IOException { 
        byte[] buf = new byte[BUF_SIZE];
        long total = 0;
        while(true) {
            int r = from.read(buf);
            if(r == -1) {
                break;
            }
            to.write(buf, 0, r);
            total += r;
        }
        return total;
    }

    public static byte[] toByteArray(BufferedInputStream in) throws IOException {
        ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
        BufferedOutputStream out = new BufferedOutputStream(bytesOut);
        copy(in, out);
        return bytesOut.toByteArray();
    }
}

Asarum

Why don't you tell us what problem you are trying to solve, instead of asking what is wrong with half the solution? – Dinar

I should have originally I realize now, but I will do that in a proper question where I simply put out what I want to achieve. – Asarum

Answer 1

Have a look at ByteArrayOutputStream: Java 7 API java.io.ByteArrayOutputStream

bytesOut = new ByteArrayOutputStream();
byte[] bytes = bytesOut.toByteArray();

Update: If you insist on doing what you are doing you can just assign the intermediate ByteArrayOutputStream to a variable and get hold of the array that way:

ByteArrayOutputStream bytesOut = new ByteArrayOutputStream()
BufferedOutputStream out = new BufferedOutputStream(bytesOut);
copy(in, out);
return bytesOut.toByteArray();

Update 2: The real question seems to be how to copy a file without reading it all into memory first:

1) Manually:

    byte[] buff = new byte[64*1024]; //or some size, can try out different sizes for performance
    BufferedInputStream in = new BufferedInputStream(new FileInputStream("fromFile"));
    BufferedOutputStream out = new BufferedOutputStream(new FileoutputStream("toFile"));
    int n = 0;
    while ((n = in.read(buff)) >= 0) {
        out.write(buff, 0, n);
    }
    in.close();
    out.close();

2) Efficiently by the OS and no loop etc:

FileChannel from = new FileInputStream(sourceFile).getChannel();
FileChanngel to = new FileOutputStream(destFile).getChannel();
to.transferFrom(from, 0, from.size());
//or from.transferTo(0, from.size(), to);
from.close();
to.close();

3) If you have Java 7 you can simplify exception and stream closing or just copy the file with the new APIs i in java 7:

java.nio.file.Files.copy(...);

see java.nio.file.Files

Gormless

You don't need any more buffering... It's building an array in memory. It doesn't get more buffered than that. – Gormless

However if you don't belieive me (you should measure yourself anyway), you can just use it anyway but assign it to it's own variable in between. I'll update the example. – Gormless

I tried to make a copy of 1.39 GB file on my computer through unbuffered streams. It gave me the "run out of memory" exception. I was told buffered streams, could be a solution to this. – Asarum

I think you or the other party misunderstood if so, that won't help. – Gormless

What you need is to not read all of it into memory first and then write it, just write it as you go. Meaning, don't use ByteArray streams. – Gormless

If all you want is to copy a file, either do it manually: read a chunk of bytes from a FileInputStream wrapped in a BufferedInputStream and write a chunk to a FileOutputSteream wrapped in a BufferedOutputStream or let the OS do it more efficiently if it is able to using transferTo() – Gormless

It's not JUST a matter of copying the file. It's supposed to be used in a web solution, kind of, where a client can send this to a server, but it's made into a byte array, before it's sent. I am still getting heap space errors, so I will update my main post. – Asarum

You might want to look at ByteBuffer instead. – Cheapen

If it is stored as a byte array inbetween it is stored in memory, and will take up 1,39GB in memory... so storing it in a byte array sounds really bad if you need any decent sized files I'm afraid. – Gormless

I posted all the code I got. I suppose I will rewrite the topics name too. Just in case. – Asarum

That code internally stores the bytes in an array for no good reason though? If that is what it is doing you can as well go with either the updated answer solution 1) or upcoming 2) – Gormless

Answer 2

DataFetcher would be perfect for this:

http://tus.svn.sourceforge.net/viewvc/tus/tjacobs/io/DataFetcher.java?revision=34&view=markup

You can use clearBuffer to clear out the buffer between reads if you are running out of memory

You'll also need the Timeout class - it's in the same project in the same package.

Escheat