How to handle file upload chunks on the server (Plupload/Spring MVC)?

Question

On our client, it is possible to be uploading large files. We want to use chunking to reduce the size of the requests. We are using Plupload, so it's easy to send up the files in chunks. However, I'm not sure how to handle the chunks. We are using Spring MVC on the server, and I current have the following for my controller method: uploadMedia(@RequestBody MultipartFile file) where MultipartFile is org.springframework.web.multipart.MultipartFile. This works just fine when not doing chunking. When I turn on the chunking on the client side, it still gets into this method just fine, but I don't see anything on the file that gets passed in that identifies what file the chunk is a part of. Hopefully I'm just missing something.

It seems like this is a common workflow, but I can't seem to find any good examples of how this is done on the server side. A solution with just Spring would be great, but if another library is needed, that is fine too. I looked some at Apache Commons FileUpload, but I couldn't find anything about chunking on there. Any help with this would be great. Thanks.

Alsatian

Answer 1

I debugged through this some more and added a HttpServletRequest to my controller method to see if there was anything available in there. I found that the request I received was a org.springframework.web.multipart.support.DefaultMultipartHttpServletRequest which has a field called multipartParameters which is just a HashMap. I found that this map had keys of name, chunks, and chunk. So, I decided to try the following as my signature:

uploadMedia(@RequestBody MultipartFile file, @RequestParam String name, @RequestParam int chunks, @RequestParam int chunk)

Sure enough, those @RequestParam parameters got populated with the name of the file, the number of chunks, and the chunk number (zero-based) respectively. Having this information with each request will make it pretty easy to assemble the chunks into the final file.

One thing to know also, is that if the file does not need to be chunked (the file size is less than the chunk size), the chunks and chunk parameters are not sent up. So, my final signature ended up looking like this:

uploadMedia(@RequestBody MultipartFile file, @RequestParam String name, @RequestParam(required=false, defaultValue="-1") int chunks, @RequestParam(required=false, defaultValue="-1") int chunk)

Then I can check for -1 to see if I need to worry about chunking at all.

So, in the controller method itself I have something like this:

Media media = new Media();
//set stuff on my Media object for storing info about the file in the DB
//....
if (chunks > 0 && chunk > 0)
{
    //Need to append the bytes in this chunk
    mediaRepository.appendBytes(media, file.getBytes());
    if (chunk == chunks - 1)
    {
        //last chunk, upload is done
        onUploadFinished(media);
    }
}
else
{
    //Write out the first set of bytes
    mediaRepository.saveBytes(media, file.getBytes());
    if (chunks <= 0)
    {
        //no chunks were needed, all the bytes have been written out, upload is done
        onUploadFinished(media);
    }
}

Alsatian

is it possible to see more of the controller? I am trying to implement the same functionality and am struggling the the rest of the controller. – Goatsucker

I've updated the answer to include more of the controller code. – Alsatian

if you don't mind can you share a portion of your mediaRepository? – Goatsucker

It's just using the java.nio.file.Files package in Java 7 (docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html). So, the methods above are pretty much one-liners. Files.write(getFilePath(media), bytes, StandardOpenOption.CREATE); and Files.write(getFilePath(media), bytes, StandardOpenOption.APPEND); – Alsatian

Are you writing the chunks of content to a temporary file before sending to the DB? – Ken

I'm not saving the bytes in the DB, just writing them to disk. There's no reason the saveBytes and appendBytes couldn't write to a DB though. What those methods do is outside the scope of this question. This question and answer shows how to get the necessary info from Plupload, and then you can do whatever you need to with that information. – Alsatian

the order of the chucks is always 0 to (chunks -1) ? So when you get the last chuck you can assume you have the earlier ones? Before you stich the chunks together do you check if you have the inidividual ones completely - meaning size of each chunk is the same except the last, or wait a few seconds till they are? just thinking of latency, threads, and hence chucks that are not yet completely saved – Frulla

It's been a while since I've worked on this project and with Plupload, but I believe the chunks are sent serially, so that it doesn't send the next chunk until it has received a response from the current chunk. And I think they always came through in order. It's possible new versions are different now, but that was my experience when using it. – Alsatian