How can you pipe an OutputStream to a StreamingDataHandler?

Question

I've got a Java web service in JAX-WS that returns an OutputStream from another method. I can't seem to figure out how to stream the OutputStream into the returned DataHandler any other way than to create a temporary file, write to it, then open it back up again as an InputStream. Here's an example:

@MTOM
@WebService
class Example {
    @WebMethod
    public @XmlMimeType("application/octet-stream") DataHandler service() {
        // Create a temporary file to write to
        File fTemp = File.createTempFile("my", "tmp");
        OutputStream out = new FileOutputStream(fTemp);

        // Method takes an output stream and writes to it
        writeToOut(out);
        out.close();

        // Create a data source and data handler based on that temporary file
        DataSource ds = new FileDataSource(fTemp);
        DataHandler dh = new DataHandler(ds);
        return dh;
    }
}

The main issue is that the writeToOut() method can return data that are far larger than the computer's memory. That's why the method is using MTOM in the first place - to stream the data. I can't seem to wrap my head around how to stream the data directly from the OutputStream that I need to provide to the returned DataHandler (and ultimately the client, who receives the StreamingDataHandler).

I've tried playing around with PipedInputStream and PipedOutputStream, but those don't seem to be quite what I need, because the DataHandler would need to be returned after the PipedOutputStream is written to.

Any ideas?

Shinleaf

See also this question – Spiderwort

Answer 1

I figured out the answer, along the lines that Christian was talking about (creating a new thread to execute writeToOut()):

@MTOM
@WebService
class Example {
    @WebMethod
    public @XmlMimeType("application/octet-stream") DataHandler service() {
        // Create piped output stream, wrap it in a final array so that the
        // OutputStream doesn't need to be finalized before sending to new Thread.
        PipedOutputStream out = new PipedOutputStream();
        InputStream in = new PipedInputStream(out);
        final Object[] args = { out };

        // Create a new thread which writes to out.
        new Thread(
            new Runnable(){
                public void run() {
                    writeToOut(args);
                    ((OutputStream)args[0]).close();
                }
            }
        ).start();

        // Return the InputStream to the client.
        DataSource ds = new ByteArrayDataSource(in, "application/octet-stream");
        DataHandler dh = new DataHandler(ds);
        return dh;
    }
}

It is a tad more complex due to final variables, but as far as I can tell this is correct. When the thread is started, it blocks when it first tries to call out.write(); at the same time, the input stream is returned to the client, who unblocks the write by reading the data. (The problem with my previous implementations of this solution was that I wasn't properly closing the stream, and thus running into errors.)

Shinleaf

I really don't know much about this, but make sure that the Pipe*Streams are either thread safe or use that "synchronised" keyword that I'm unfamiliar with. – Bastille

Please note that the javadoc of javax.mail.util.ByteArrayDataSource states that the InputStream is read to memory completely on construction. This may result in an OutOfMemoryError when dealing with large files – Spiderwort

Answer 2

Sorry, I only did this for C# and not java, but I think your method should launch a thread to run "writeToOut(out);" in parralel. You need to create a special stream and pass it to the new thread which gives that stream to writeToOut. After starting the thread you return that stream-object to your caller.

If you only have a method that writes to a stream and returns afterwards and another method that consumes a stream and returns afterwards, there is no other way.

Of coure the tricky part is to get hold of such a -multithreading safe- stream: It shall block each side if an internal buffer is too full.

Don't know if a Java-pipe-stream works for that.

Bastille

+1 - The idea is correct, and I thank you for putting me on the right path, but my answer has the actual Java solution that I want future people to be able to find first if they come across the same problem. – Shinleaf

Answer 3

Wrapper pattern ? :-).

Custom javax.activation.DataSource implementation (only 4 methods) to be able to do this ?

return new DataHandler(new DataSource() { 
  // implement getOutputStream to return the stream used inside writeToOut() 
  ... 
});   

I don't have the IDE available to test this so i'm only doing a suggestion. I would also need the writeToOut general layout :-).

Mayhap

A working implementation can be found in this answer. – Spiderwort

Answer 4

In my application I use InputStreamDataSource implementation that take InputStream as constructor argument instead of File in FileDataSource. It works so far.

public class InputStreamDataSource implements DataSource {

ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private final String name;

public InputStreamDataSource(InputStream inputStream, String name) {
    this.name = name;
    try {
        int nRead;
        byte[] data = new byte[16384];
        while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
            buffer.write(data, 0, nRead);
        }

        buffer.flush();
        inputStream.close();
    } catch (IOException e) {
        e.printStackTrace();
    }

}

@Override
public String getContentType() {
    return new MimetypesFileTypeMap().getContentType(name);
}

@Override
public InputStream getInputStream() throws IOException {
    return new ByteArrayInputStream(buffer.toByteArray());
}

@Override
public String getName() {
    return name;
}

@Override
public OutputStream getOutputStream() throws IOException {
    throw new IOException("Read-only data");
}

}

Yurikoyursa