Send large files to GAE Blobstore via API (from S3)

Question

I'm trying to send large files (50MB-2GB) that I have stored in S3 using filepicker.io to the Google App Engine Blobstore (in Python). I would like this to happen without going through a form on the client browser as it defeats the project requirements (and often hangs with very large files).

I tried multiple solutions, including:

• trying to load the file into GAE with urlfetch (but GAE has a 32MB limit for requests/responses)
• constructing a multi-part form in python and sending it to blobstore.create_upload_url() (can't transfer the file via url, and can't load it in the script because of the 32MB limit)
• using boto to read the file straight into the blobstore (connection times out and logs show encountered HTTPException exception from boto that triggers CancelledError: The API call logservice.Flush() was explicitly cancelled. from GAE that crashes the process.

I am struggling to find a working solution. Any hints on how I could perform this transfer, or how to pass the file from s3 as a form attachment without loading it in python first (ie. just specifying its url) would be very much appreciated.

Polymerism

use the http libraries, not a full answer but check it out – Warplane

@peterretief you mean something like python requests? – Polymerism

I posted a question related to this suggestion here – Polymerism

Answer 1

The BlobstoreUploadHandler isn't constrained by a 32MB limit: https://developers.google.com/appengine/docs/python/tools/webapp/blobstorehandlers. However, I'm not sure how this might fit into your app. If you can post the file to an endpoint handled by a BlobstoreUploadHandler, then you should be good to go.

Amato

thanks Robert, I have a BlobstoreUploadHandler already. My problem is how to send it the very large file that I have stored on S3 - I can't find a way to attach it to a POST request without loading it elsewhere in GAE first (which will cross the 32MB limit). Hope this makes sense. – Polymerism

There are those backend processors that don't have any limitations at all. Have you looked into those? It is an interesting problem. I suppose you can't just leave the files in S3. If you are letting the user interact with the app through the browser then you can use the html5 File api: html5rocks.com/en/tutorials/file/dndfiles/#toc-slicing-files to load the file into the browser (what happens when you load a 2GB file though). Otherwise, you'll have to run a process outside the app engine to do all this. – Amato