How to clone an InputStream?
Asked Answered
A

10

201

I have a InputStream that I pass to a method to do some processing. I will use the same InputStream in other method, but after the first processing, the InputStream appears be closed inside the method.

How I can clone the InputStream to send to the method that closes him? There is another solution?

EDIT: the methods that closes the InputStream is an external method from a lib. I dont have control about closing or not.

private String getContent(HttpURLConnection con) {
    InputStream content = null;
    String charset = "";
    try {
        content = con.getInputStream();
        CloseShieldInputStream csContent = new CloseShieldInputStream(content);
        charset = getCharset(csContent);            
        return  IOUtils.toString(content,charset);
    } catch (Exception e) {
        System.out.println("Error downloading page: " + e);
        return null;
    }
}

private String getCharset(InputStream content) {
    try {
        Source parser = new Source(content);
        return parser.getEncoding();
    } catch (Exception e) {
        System.out.println("Error determining charset: " + e);
        return "UTF-8";
    }
}
Anamariaanamnesis answered 7/5, 2011 at 20:34 Comment(4)
Do you want to "reset" the stream after the method has returned? I.e., read the stream from the beginning?Plasticity
Yes, the methods that closes the InputStream returns the charset it was encoded. The second method is to convert the InputStream to a String using the charset found in the first method.Anamariaanamnesis
You should in that case be able to do what I'm describing in my answer.Lornalorne
I dont know the best way to resolve it, but I resolve my problem otherwise. The method toString of the Jericho HTML Parser returns the String formatted in the correct format. It's all I need at moment.Anamariaanamnesis
R
245

If all you want to do is read the same information more than once, and the input data is small enough to fit into memory, you can copy the data from your InputStream to a ByteArrayOutputStream.

Then you can obtain the associated array of bytes and open as many "cloned" ByteArrayInputStreams as you like.

ByteArrayOutputStream baos = new ByteArrayOutputStream();

// Code simulating the copy
// You could alternatively use NIO
// And please, unlike me, do something about the Exceptions :D
byte[] buffer = new byte[1024];
int len;
while ((len = input.read(buffer)) > -1 ) {
    baos.write(buffer, 0, len);
}
baos.flush();
    
// Open new InputStreams using recorded bytes
// Can be repeated as many times as you wish
InputStream is1 = new ByteArrayInputStream(baos.toByteArray()); 
InputStream is2 = new ByteArrayInputStream(baos.toByteArray()); 

But if you really need to keep the original stream open to receive new data, then you will need to track the external call to close(). You will need to prevent close() from being called somehow.

UPDATE (2019):

Since Java 9 the the middle bits can be replaced with InputStream.transferTo:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
input.transferTo(baos);
InputStream firstClone = new ByteArrayInputStream(baos.toByteArray()); 
InputStream secondClone = new ByteArrayInputStream(baos.toByteArray()); 
Randolphrandom answered 7/5, 2011 at 21:32 Comment(7)
I fount another solution to my problem thar not involves copying the InputStream, but I think if I need copy the InputStream, this is the best solution.Anamariaanamnesis
This approach consumes memory proportional to the full content of the input stream. Better to use TeeInputStream as described in the answer over here.Plasticity
IOUtils (from apache commons) has a copy method which would do the buffer read/write in the middle of your code.Sematic
@Plasticity sorry to ask about such an old comment, but did you test the memory consumption, and if so, were your streams consuming at more or less the same time? The proposed solution (linked in your comment) uses BufferedReader, which is backed by an array, same as the ByteArrayOutputStream used in the answer above.Toneme
I'm not seeing how using TeeInputStream would improve memory consumption, except for the fact that the answer above uses a second call to baos.toByteArray() instead of using PipedInputStream for the second copyToneme
If you do have Java 9, why not use ByteArrayInputStream(input.readAllBytes())?Megawatt
@PhilippeGioseffi, InputStream.readAllBytes will return an empty byte array after the first invocation (hence the need for an intermediate byte[] or ByteArrayOutputStream). On top of that ByteArrayOutputStream.toByteArray returns a copy of the original byte array, which may or may not be desirable (the code above creates independent stream "clones").Randolphrandom
R
36

You want to use Apache's CloseShieldInputStream:

This is a wrapper that will prevent the stream from being closed. You'd do something like this.

InputStream is = null;

is = getStream(); //obtain the stream 
CloseShieldInputStream csis = new CloseShieldInputStream(is);

// call the bad function that does things it shouldn't
badFunction(csis);

// happiness follows: do something with the original input stream
is.read();
Respectively answered 7/5, 2011 at 21:9 Comment(9)
Looks good, but dont works here. I will edit my post with the code.Anamariaanamnesis
CloseShield isn't working because your original HttpURLConnection input stream is beeing closed somewhere. Shouldn't your method call IOUtils with the protected stream IOUtils.toString(csContent,charset)?Randolphrandom
Maybe can be this. I can prevent from the HttpURLConnection be closed?Anamariaanamnesis
What is the Source class doing? Is it the call that is closing the stream?Respectively
@Renato. Maybe the problem is not the close() call at all, but the fact the Stream is being read to the end. Since mark() and reset() may not be the best methods for http connections, maybe you should take a look at the byte array approach described at my answer.Randolphrandom
One more thing, you can always open a new connection to the same URL. See here: #5807840Randolphrandom
For few inputstreams mark/reset is not supported such as InfalterInputStreamCopland
How to close this stream if it will not be used anymore?Isom
CloseShieldInputStream constructor is now deprecated. You can use static method CloseShieldInputStream.wrap(is) instead.Upstanding
L
13

You can't clone it, and how you are going to solve your problem depends on what the source of the data is.

One solution is to read all data from the InputStream into a byte array, and then create a ByteArrayInputStream around that byte array, and pass that input stream into your method.

Edit 1: That is, if the other method also needs to read the same data. I.e you want to "reset" the stream.

Lornalorne answered 7/5, 2011 at 20:40 Comment(1)
I don't know what part you need help with. I guess you know how to read from a stream? Read all data from the InputStream, and write the data to ByteArrayOutputStream. Call toByteArray() on the ByteArrayOutputStream after you have completed reading all data. Then pass that byte array into the constructor of a ByteArrayInputStream.Lornalorne
M
11

If the data read from the stream is large, I would recommend using a TeeInputStream from Apache Commons IO. That way you can essentially replicate the input and pass a t'd pipe as your clone.

Myrtia answered 7/5, 2011 at 21:34 Comment(1)
No code answer....Chisholm
J
6

This might not work in all situations, but here is what I did: I extended the FilterInputStream class and do the required processing of the bytes as the external lib reads the data.

public class StreamBytesWithExtraProcessingInputStream extends FilterInputStream {

    protected StreamBytesWithExtraProcessingInputStream(InputStream in) {
        super(in);
    }

    @Override
    public int read() throws IOException {
        int readByte = super.read();
        processByte(readByte);
        return readByte;
    }

    @Override
    public int read(byte[] buffer, int offset, int count) throws IOException {
        int readBytes = super.read(buffer, offset, count);
        processBytes(buffer, offset, readBytes);
        return readBytes;
    }

    private void processBytes(byte[] buffer, int offset, int readBytes) {
       for (int i = 0; i < readBytes; i++) {
           processByte(buffer[i + offset]);
       }
    }

    private void processByte(int readByte) {
       // TODO do processing here
    }

}

Then you simply pass an instance of StreamBytesWithExtraProcessingInputStream where you would have passed in the input stream. With the original input stream as constructor parameter.

It should be noted that this works byte for byte, so don't use this if high performance is a requirement.

Jews answered 7/3, 2013 at 10:7 Comment(0)
K
6

UPD. Check the comment before. It isn't exactly what was asked.

If you are using apache.commons you may copy streams using IOUtils .

You can use following code:

InputStream = IOUtils.toBufferedInputStream(toCopy);

Here is the full example suitable for your situation:

public void cloneStream() throws IOException{
    InputStream toCopy=IOUtils.toInputStream("aaa");
    InputStream dest= null;
    dest=IOUtils.toBufferedInputStream(toCopy);
    toCopy.close();
    String result = new String(IOUtils.toByteArray(dest));
    System.out.println(result);
}

This code requires some dependencies:

MAVEN

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.4</version>
</dependency>

GRADLE

'commons-io:commons-io:2.4'

Here is the DOC reference for this method:

Fetches entire contents of an InputStream and represent same data as result InputStream. This method is useful where,

Source InputStream is slow. It has network resources associated, so we cannot keep it open for long time. It has network timeout associated.

You can find more about IOUtils here: http://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/IOUtils.html#toBufferedInputStream(java.io.InputStream)

Khania answered 11/2, 2016 at 12:14 Comment(1)
This does not clone the input stream but only buffers it. That's not the same; the OP wants to re-read (a copy of) the same stream.Remonstrant
C
3

Below is the solution with Kotlin.

You can copy your InputStream into ByteArray

val inputStream = ...

val byteOutputStream = ByteArrayOutputStream()
inputStream.use { input ->
    byteOutputStream.use { output ->
        input.copyTo(output)
    }
}

val byteInputStream = ByteArrayInputStream(byteOutputStream.toByteArray())

If you need to read the byteInputStream multiple times, call byteInputStream.reset() before reading again.

https://code.luasoftware.com/tutorials/kotlin/how-to-clone-inputstream/

Chondroma answered 30/4, 2018 at 4:3 Comment(0)
A
1

Cloning an input stream might not be a good idea, because this requires deep knowledge about the details of the input stream being cloned. A workaround for this is to create a new input stream that reads from the same source again.

So using some Java 8 features this would look like this:

public class Foo {

    private Supplier<InputStream> inputStreamSupplier;

    public void bar() {
        procesDataThisWay(inputStreamSupplier.get());
        procesDataTheOtherWay(inputStreamSupplier.get());
    }

    private void procesDataThisWay(InputStream) {
        // ...
    }

    private void procesDataTheOtherWay(InputStream) {
        // ...
    }
}

This method has the positive effect that it will reuse code that is already in place - the creation of the input stream encapsulated in inputStreamSupplier. And there is no need to maintain a second code path for the cloning of the stream.

On the other hand, if reading from the stream is expensive (because a it's done over a low bandwith connection), then this method will double the costs. This could be circumvented by using a specific supplier that will store the stream content locally first and provide an InputStream for that now local resource.

Aretino answered 15/2, 2016 at 8:35 Comment(3)
This answer is not clear to me. How do you initialise the supplier from an existing is?Cherie
@Cherie As I wrote Cloning an input stream might not be a good idea, because this requires deep knowledge about the details of the input stream being cloned. you can't use the supplier to create an input stream fron an existing one. The supplier could use a java.io.File or java.net.URL for example to create a new input stream each time it is invoked.Aretino
I see now. This will not work with inputstream as the OP explicitly asks, but with File or URL if they are the original source of data. ThanksCherie
B
0

The class below should do the trick. Just create an instance, call the "multiply" method, and provide the source input stream and the amount of duplicates you need.

Important: you must consume all cloned streams simultaneously in separate threads.

package foo.bar;

import java.io.IOException;
import java.io.InputStream;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class InputStreamMultiplier {
    protected static final int BUFFER_SIZE = 1024;
    private ExecutorService executorService = Executors.newCachedThreadPool();

    public InputStream[] multiply(final InputStream source, int count) throws IOException {
        PipedInputStream[] ins = new PipedInputStream[count];
        final PipedOutputStream[] outs = new PipedOutputStream[count];

        for (int i = 0; i < count; i++)
        {
            ins[i] = new PipedInputStream();
            outs[i] = new PipedOutputStream(ins[i]);
        }

        executorService.execute(new Runnable() {
            public void run() {
                try {
                    copy(source, outs);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        });

        return ins;
    }

    protected void copy(final InputStream source, final PipedOutputStream[] outs) throws IOException {
        byte[] buffer = new byte[BUFFER_SIZE];
        int n = 0;
        try {
            while (-1 != (n = source.read(buffer))) {
                //write each chunk to all output streams
                for (PipedOutputStream out : outs) {
                    out.write(buffer, 0, n);
                }
            }
        } finally {
            //close all output streams
            for (PipedOutputStream out : outs) {
                try {
                    out.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}
Bradford answered 30/9, 2014 at 22:22 Comment(1)
Doesn't answer the question. He wants to use the stream in one method to determine the charset and then re-read it along with its charset in a second method.Pleurisy
R
0

Enhancing the @Anthony Accioly with the example.

InputStream: Clones the bytes-Stream and provides number of copies as a List Collection.

public static List<InputStream> multiplyBytes(InputStream input, int cloneCount) throws IOException {
    List<InputStream> copies = new ArrayList<InputStream>();
    
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    copy(input, baos);
    
    for (int i = 0; i < cloneCount; i++) {
        copies.add(new ByteArrayInputStream(baos.toByteArray()));
    }
    return copies;
}
// IOException - If reading the Reader or Writing into the Writer goes wrong.
public static void copy(Reader in, Writer out) throws IOException {
    try {
        char[] buffer = new char[1024];
        int nrOfBytes = -1;
        while ((nrOfBytes = in.read(buffer)) != -1) {
            out.write(buffer, 0, nrOfBytes);
        }
        out.flush();
    } finally {
        close(in);
        close(out);
    }
}

Reader: Clones the chars-Stream and provides number of copies as a List Collection.

public static List<Reader> multiplyChars(Reader reader, int cloneCOunt) throws IOException {
    List<Reader> copies = new ArrayList<Reader>();
    BufferedReader bufferedInput = new BufferedReader(reader);
    StringBuffer buffer = new StringBuffer();
    String delimiter = System.getProperty("line.separator");
    String line;
    while ((line = bufferedInput.readLine()) != null) {
        if (!buffer.toString().equals(""))
            buffer.append(delimiter);
        buffer.append(line);
    }
    close(bufferedInput);
    for (int i = 0; i < cloneCOunt; i++) {
        copies.add(new StringReader(buffer.toString()));
    }
    return copies;
}
public static void copy(InputStream in, OutputStream out) throws IOException {
    try {
        byte[] buffer = new byte[1024];
        int nrOfBytes = -1;
        while ((nrOfBytes = in.read(buffer)) != -1) {
            out.write(buffer, 0, nrOfBytes);
        }
        out.flush();
    } finally {
        close(in);
        close(out);
    }
}

Full Example:

public class SampleTest {

    public static void main(String[] args) throws IOException {
        String filePath = "C:/Yash/StackoverflowSSL.cer";
        InputStream fileStream = new FileInputStream(new File(filePath) );
        
        List<InputStream> bytesCopy = multiplyBytes(fileStream, 3);
        for (Iterator<InputStream> iterator = bytesCopy.iterator(); iterator.hasNext();) {
            InputStream inputStream = (InputStream) iterator.next();
            System.out.println("Byte Stream:"+ inputStream.available()); // Byte Stream:1784
        }
        printInputStream(bytesCopy.get(0));
        
        //java.sql.Clob clob = ((Clob) getValue(sql)); - clob.getCharacterStream();
        Reader stringReader = new StringReader("StringReader that reads Characters from the specified string.");
        List<Reader> charsCopy = multiplyChars(stringReader, 3);
        for (Iterator<Reader> iterator = charsCopy.iterator(); iterator.hasNext();) {
            Reader reader = (Reader) iterator.next();
            System.out.println("Chars Stream:"+reader.read()); // Chars Stream:83
        }
        printReader(charsCopy.get(0));
    }
    
    // Reader, InputStream - Prints the contents of the reader to System.out.
    public static void printReader(Reader reader) throws IOException {
        BufferedReader br = new BufferedReader(reader);
        String s;
        while ((s = br.readLine()) != null) {
            System.out.println(s);
        }
    }
    public static void printInputStream(InputStream inputStream) throws IOException {
        printReader(new InputStreamReader(inputStream));
    }

    // Closes an opened resource, catching any exceptions.
    public static void close(Closeable resource) {
        if (resource != null) {
            try {
                resource.close();
            } catch (IOException e) {
                System.err.println(e);
            }
        }
    }
}

Revivify answered 1/6, 2021 at 8:24 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.