Process Uploaded file on web server without storing locally first?

Question

I am trying to process the user uploaded file real time on the websever, but it seems, APACHE invokes PHP, only once complete file is uploaded.

When i uploaded the file using CURL, and set

Transfer-Encoding : "Chunked"

I had some success, but can't do same thing via browser.

• I used Dropzone.js but when i tried to set same header, it said Transfer -Encoding is an unsafe header, hence not setting it.

This answer explains what is the issue there. Can't set Transfer-Encoding :"Chunked from Browser"

In a Nutshell problem is , when a user uploads the file to webserver, i want webserver to start processing it as soon as first byte is available. by process i mean, PIPING it to a Named Pipe.

Dont want 500mb first getting uploaded to a server, then start processing it.

But with current Webserver (APACHE - PHP), I cant seem to be able to accomplish it.

could someone please explain, what technology stack or workarounds to use, so that i can upload the large file via browser and start processing it, as soon as first byte is available.

Ginni

you can do it via node js if you interested I can post answer – Tegan

Can you upload to Amazon AWS in chunk of 5 MB using fineuploader and initiate one cron to check new AWS file and process it for Named Pipe? – Mintz

@Tegan : yes please post, currently i am trying to do same with busboy library, on 'data' function i am piping the chunk to child process..! – Ginni

@ChetanPatel : i dont intend to stream to AWS, i want to stream to an external app, for example tshark. – Ginni

Answer 1

It is possible to use NodeJS/Multiparty to do that. Here they have an example of a direct upload to Amazon S3. This is the form, which sets content type to multipart/form-data. And here is the function for form parts processing. part parameter is of type ReadableStream, which will allow per-chunk processing of the input using data event.

More on readable streams in node js is here.

Brainy

Answer 2

If you really want that (sorry don`t think thats a good idea) you should try looking for a FUSE Filesystem which does your job.

Maybe there is already one https://github.com/libfuse/libfuse/wiki/Filesystems

Or you should write your own.

But remember as soon as the upload is completed and the post script finishes his job the temp file will be deleted

Loblolly

I am afraid , this is not what i am looking for. i just want to stream the uploaded file to an external app, without storing it first. – Ginni

Answer 3

you can upload file with html5 resumable upload tools (like Resumable.js) and process uploaded parts as soon as they received.

or as a workaround , you may find the path of uploaded file (usually in /tmp) and then write a background job to stream it to 3rd app. it may be harder.

there may be other solutions...

Prearrange