Filepicker? Upload big files via HTML5 to S3 with no backend

Question

Upload files with multipart/form-data is straight forward and works well most of time until you started to be focused on big files upload. If we look closely what happens during a file upload:

• client sends POST request with the file content in BODY

• webserver accepts the request and initiates data transfer (or returns error 413 if the file size is exceed the limit)

• webserver starts to populate buffers (depends on file and buffers size), store it on disk and send it via socket/network to back-end

• back-end verifies the authentication (take a look, once file is uploaded)

• back-end reads the file and cuts few headers Content-Disposition, Content-Type, stores it on disk again back-end performs all you need to do with the file

To avoid such overhead we dump file on disk (Nginx client_body_in_file_only) and manage the callback to be send further down the line. Then queue worker picks the file up and do what required. It works for inter-server communication pretty slick but we have to solve similar problem with client side upload.

We also have client-side S3 upload solution. No back-end interaction happens. For video upload we manage the video to convert to the format h.264 Baseline / AAC with Zencoder.

Currently we use modified Flash uploader based on s3-swf-upload-plugin with combination of Zencoder JS SDK which is really efficient but uses Flash.

Question. How to reach the same goal with HTML5 file uploader? Does Filepicker.io and Zencoder solve the problem? What is the recommended way to manage HTML5 file upload with no back-end interaction?

The requirements are the following:

• HTML5, not flash
• to upload video with post-processing to make it compatible with HTML5 players and mobile
• to upload images wtih post-processing (resize, crop, rotate)
• to upload documents like PDF with a preview functionality

Does https://www.filepicker.com make a good job?

Jaquelynjaquenetta

I have to research a bit more, but wouldn't client_body_in_file_only cause more disk access and thus decreased performance? The Nginx docs say it should be used for debugging primarily. – Aerophyte

@Aerophyte no, it works in production many years for us, all is perfect. I talked to Nginx core team developers, they confirmed it is pretty stable for production work load. – Jaquelynjaquenetta

Answer 1

I'm using filepicker for 2 years now, and without doubt it's worth the price. don't try to manage file upload (from google drive, from ios, from my camera, from dropbox...) Filepicker handles that very well and provide you a ready to use url. Spend more time working on your core business, file upload is really easy to delegate

Wideangle

Answer 2

To upload a big files to S3 there is a REST API for Multipart Upload, which works the following way

1. initiate an upload
2. upload file content split into multiple requests
3. finish the upload

the API is also available for calling from javascript and the uploaded file can be split to multiple requests using File/Blob slice API

The only problem is that to be able to authenticate to S3 from javascript you need to pass your authentication details. This is usually solved by some interlayer like PHP so the authentication details are not stored in javascript files.

Similar question on SO: HTML5 and Amazon S3 Multi-Part uploads

EDIT

• image operations like cropping and resizing can be done by canvas. Just load the local image into the canvas element, make the edits you need and then you can generate jpeg/png image data stream using canvas.toDataUrl method.
• PDF preview is possible, there is a PDF.js lib that can render local PDF file into the canvas
• AFAIK there is no way to do video conversion on the client side

Landscapist

It does solve but only one particular problem with client side upload, what about everything else? – Jaquelynjaquenetta

just added some info about image cropping and PDF preview – Landscapist