HTML5 and Amazon S3 Multi-Part uploads

Question

Is it possible to use the HTML 5 File API (for example, this library: https://github.com/23/resumable.js ) in conjunction with the S3 multi-part upload feature?

http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

Sulphurate

Answer 1

Yes, but you will need some kind of server backend to handle the Amazon API keys in a more secure way if you are going to make it part of a public website.

You can find what looks like a complete example implementation of this these projects:

• s3-multipart-upload-browser which uses a PHP backend
• s3_multipart which uses Ruby.

Please note that I have not used, tested or reviewed these projects.

A rough description of the sequence is as follows:

1. User
   • loads webpage
   • selects file to upload
   • hits upload button
2. Webpage
   • sends info about file to server
3. Server
   • creates multipart upload with Amazon API
   • sends "key"(filename) and "upload id" back to Webpage
4. Webpage
   • works out size of parts
   • requests Server to sign part passing "key", "upload id", part info
5. Server
   • signs a part request, sends "part upload url", "date" and "auth header"
6. Webpage
   • sends part data directly to Amazon S3 via "part upload url" using "date" and "auth header"
   • keeps track of part ids
7. Server & Webpage
   • repeats 5 & 6 for each additional part, resuming if required
8. Webpage
   • makes "upload complete" request to server (passing all the part info)
9. Server
   • makes request to Amazon API to complete the creation of file
10. Webpage
    • inform user of error or success

Notes:

• If upload is aborted, this must also be handled server side else the parts / uploads started will be left to take up space in the S3 Bucket.
• It might take a few minutes to complete the "upload complete" request to Amazon.

Ebonee

Thank you for the detailed answer! I've actually started implementing the protocol you detailed before, but haven't gotten around to actually finishing it yet: github.com/keichan34/s3uploader/tree/2.0-wip – Sulphurate

Great, I've noticed you are using Ruby in your project and also found another Ruby project which I have added to the answer for your reference. – Ebonee

Here is another example in PHP https://github.com/ienzam/s3-multipart-upload-browser – Spacing

@BausTheBig this is/was the first project linked in the answer. Obviously it wasn't clear so thanks to your comment I have updated the answer to make the link more clear. – Ebonee

Great thanks! The answer is very detailed, exactly what I needed right now. This is another interesting project (it uses Python) to look at: github.com/cinely/mule-uploader#mule-upload – Spacing

@BausTheBig actually, the bulk of the project is written in Javascript, it only needs a small server-side endpoint, which can be written in any language easily. The "canonical" endpoint is written in Python though (Flask). – Andreandrea