Is is possible to get an email attachment into App Engine's Blobstore?

Question

I've managed to get email attachments onto Amazon S3 from a GAE incoming email, but does anyone know a technique to get an attachment, like an image, into the blobstore.

Any help would be much appreciated.

Code so far (with help from Alex)

upload_url = blobstore.create_upload_url('/upload')
msg = MIMEMultipart()
msg.set_type('multipart/form-data')
msg.set_payload({'file': content})
result = urlfetch.fetch(upload_url, payload=urllib.urlencode(msg), method=urlfetch.POST, headers={'Content-Type': 'multipart/form-data'})

Emersion

Is there a reason you can't just store the uploaded file directly in the datastore? – Neckerchief

The upload field on the page can accept files bigger than 1MB, I want all the files to go into the blobstore, even from emails so I need the email attachments to go in there too. – Emersion

Answer 1

App Engine (version 1.4.3) allows you to directly write data to the blobstore.
You no longer need to use the upload url method.

Handbill

Just to save everyone's time - here is new link to the same article - developers.google.com/appengine/docs/python/blobstore/… – Newspaperman

Answer 2

To receive mail in your GAE app, follow the docs here: in particular, you'll get a instance of class InboundEmailMessage with an attachments attribute which, and I quote:

is a list of file attachments, possibly empty. Each value in the list is a tuple of two elements: the filename and the file contents.

Then, per these GAE docs, you "create an upload URL" and in your upload handler (typically a subclass of BlobstoreUploadHandler) you use get_upload to get BlobInfo instances and put their metadata somewhere that will later let you fetch them back as your app may require.

Finally, you POST the data (that you have from attachments, above) to your own freshly generated "upload URL", e.g. using urlfetch.fetch (with method-POST and a payload in standard application/x-www-form-urlencoded encoding for the "form" that the user would be filling in if they were uploading the data directly, which is the "normal" way to put data in the blobstore -- e.g. you can use urllib.urlencode to prepare the payload).

That "self-POST" will be using another instance of your app to "receive" the data into the blobstore (while the instance that received the email waits, as fetch is synchronous).

Bathysphere

Wouldn't you need a multipart encoded form request (as thee's a file attachment there) ? I've been trying to code something that'll post to blobstore from server code for some time now without success.... – Deitz

Yes, I do believe you need multipart (and shd have mentioned that the email module can help preparing that). – Bathysphere

@Tomás, I don't have working code because I don't have blobstore in my GAE account (as I haven't enabled billing there). For the encoding, having used email, use msg.flatten(). I suggest you edit your Q to show this code so you can format it. – Bathysphere

I've tried a whole manner of things now Alex, to no avail. I don't know enough about what to encode, how to encode it and when. Are you sure that these procedure is even possible? – Emersion

@Tomás, I'm sure it's possible, I'm not sure on all details -- I'll try to find the time to develop a full solution but it's hard without actual blobstore access, as I mentioned. – Bathysphere

Happy to add you to my list of developers, feel free to upload any solution you like, still very much in the testing stage. – Emersion

Added you to Developer list Alex, would really appreciate any demonstration of Image Attachment to Blobstore, think it would really benefit the GAE community. Feel free to erase any solution currently live. – Emersion

@Tomás, I'm working on a simple variant of the "blobstore helloworld" at code.google.com/appengine/docs/python/blobstore/overview.html which also allows you to fake the upload with a post, trying to leverage code.activestate.com/recipes/… -- just so you know what I'm up to;-) – Bathysphere

I've got a simplistic prototype up at aleax.it/blosto.zip but I can't try it on my local SDK, since in the SDK any "post to the same app" just times out (I've verified that with a trivial post that easily works to any other app). That's quite a bother as it makes it unfeasible to try on the local SDK any app that "posts to itself" (even though it would work once uploaded since a new instance of the app is involved in that case:-(. But you might want to look at how the multipart post body's constructed -- that part should work. – Bathysphere

Alex, thanks for posting that code. I tried your approach and it worked great for attachments that are under 1 MB. However, urlfetch is failing when processing attachments > 1 MB. This is problematic because what I'm trying to do is process emailed photos and shrink them down to a size I can store in the datastore. I'm able to do this without a problem using traditional html forms / blobstore, but when trying your code for emailed attachements I get "call urlfetch.Fetch() was too large." Any ideas? – Arbitrary

@ryan, unfortunately I have no idea of how to work around this limitation (which is not in the blobstore but in urlfetch). I guess it needs a new feature request on App Engine's tracker (to let the app eng team know about the need for attachment->blobstore "fast path" data flow). – Bathysphere

As of Dec 10 2010, the urlfetch limitation on GAE is 32MB (see code.google.com/p/googleappengine/issues/detail?id=1355) – Cabe

I think the urlfetch limitation is 32MB response size. The limit we're running into here is the 1MB request size. – Advertisement