Building large MTOM/XOP messages with JAX-WS

Question

I have a question about using MTOM/XOP with JAX-WS. I'm writing a web service which sends large amounts of binary data. The client requests a number of files and the server returns the files in the response.

I'm able to get it to build the response correctly so that it correctly implements XOP, but I run into memory-related issues becasuse it stores the entire response in memory before sending it. The files this web service sends can get very large (like, giga-bytes large), so storing the response in memory is not an option.

This Oracle website (and along with this one) seems to solve this problem, but I just don't understand it. I think they use a DataHandler object to stream the request/response, but I can't figure out how they instantiate it.

I'm generating my JAX-WS class files from an existing WSDL using wsimport. I'm using JAX-WS RI 2.1.6 with Java 6.

How do I send the response as I'm building it without having to store in all in memory first?

Thanks in advance for your help.

---

UPDATE 12/17: I added the following attributes to the schema element in the WSDL that holds the binary data. This causes wsimport to add a DataHandler object to the JAXB class. A FileDataHandler can then be added to the response, instead of adding the entire contents of the file, allowing the server to stream the contents of each file, instead of holding them all in memory:

xmlns:xmime="http://www.w3.org/2005/05/xmlmime" 
xmime:expectedContentTypes="application/octet-stream"

So, the server correctly builds the response now, and client properly saves each file to disk when it receives the request. However, the client still reads the entire response into memory for some reason.

---

The server code (SIB):

@MTOM
@StreamingAttachment(parseEagerly = true, memoryThreshold = 4000000L) 
@WebService(...)
public class DownloadFilesPortTypeImpl implements DownloadFilesPortType {
 @Override
 public FileSetResponseType downloadFileSet(FileSetRequestType body) {
        FileSetResponseType response = new FileSetResponseType();
        for (FileRequest freq : body.getFileRequest()){
            try{
                //find the file on disk
                File file = findFile(freq.getFileId());

                //read the file data into memory
                byte[] fileData;
                {
                    FileInputStream in = new FileInputStream(file);
                    ByteArrayOutputStream out = new ByteArrayOutputStream();
                    byte buf[] = new byte[8192];
                    int read;
                    while ((read = in.read(buf)) != -1){
                         out.write(buf, 0, read);
                    }
                    in.close();
                    out.close();
                    fileData = out.toByteArray();
                }

                //add the file to the response
                FileResponse fresp = new FileResponse();
                fresp.setFileId(freq.getFileId());
                fresp.setData(fileData); //<-- type "xs:base64Binary"
                response.getFileResponse().add(fresp);
            }
            catch (IOException e){
            }
        }

        return response;
 }
}

The client code:

DownloadFilesService service = new DownloadFilesService();
MTOMFeature mtomFeature = new MTOMFeature();
StreamingAttachmentFeature stf = new StreamingAttachmentFeature(null, true, 4000000L);
DownloadFilesPortType port = service.getDownloadFilesPortSoap12(mtomFeature, stf);

FileSetRequestType request = new FileSetRequestType();

FileRequest freq = new FileRequest();
freq.setFileId("1234");
request.getFileRequest().add(freq);

freq = new FileRequest();
freq.setFileId("9876");
request.getFileRequest().add(freq);

//...

FileSetResponseType response = port.downloadFileSet(request); //reads entire response into memory
for (FileResponse fres : response.getFileResponse()){
    byte[] data = fres.getFileData();
    //...
}

Chirk

Answer 1

You make your own class that implements DataSource and construct the DataHandler passing it in. It can even be anonymous.

In Apache CXF, we make it much easier to do this. You can just have a 'getter' that returns a DataSource or a DataHandler. The elaborate scheme in the code you've posted is not something familiar to me.

I think that the same methods work with the JDK's JAX-WS+JAXB. See this.

Cirrocumulus

Where would I put the DataSource? Would I have to add a parameter to the SEI/SIB "downloadFileSet" methods (which I'd rather not do because the SEI was auto-generated by wsimport)? – Chirk

Thanks for the link. I added "xmime:expectedContentTypes" to the appropriate element in the WSDL types section, which caused wsimport to create a DataHandler object in the JAX bean. The server is creating the response OK, but I'm still getting an OutOfMemoryError exception on the client side. I checked my temp folder and it is correctly saving all files over 4MB there (as specified in the "StreamingAttachmentFeature" object)...so it's like it's writing them out to disk AND storing them in memory. – Chirk

Now you are in the territory of the specific behavior of Sun's reference implementation. Maybe you should open a new question and someone will help. Or you could use CXF :-) – Cirrocumulus

Ugh...I got this far, I'd hate to switch implementation now, but maybe I'll have to. – Chirk

Answer 2

With JAX-WS RI included in Java 6 (Metro or part of it), the client code must set HTTP chunking size to enable streaming of large MTOM attachments through DataHandler InputStream.

((BindingProvider)port).getRequestContext()
   .put(JAXWSProperties.HTTP_CLIENT_STREAMING_CHUNK_SIZE, 8192);

BUT at the moment it just not works because of a bug JAX_WS-936

As a result, even with proper chunking and DataHandler usage, an attachment gets fully loaded in memory immediately at service invocation. I have checked network traffic: full content transfert before reading DataHandler input stream, and Java heap memory usage: it has increased to contain at least three copies of the attachment.

Frampton