How to mix Django, Uploadify, and S3Boto Storage Backend?

Question

Background

I'm doing fairly big file uploads on Django. File size is generally 10MB-100MB.

I'm on Heroku and I've been hitting the request timeout of 30 seconds.

The Beginning

In order to get around the limit, Heroku's recommendation is to upload from the browser DIRECTLY to S3.

Amazon documents this by showing you how to write an HTML form to perform the upload.

Since I'm on Django, rather than write the HTML by hand, I'm using django-uploadify-s3 (example). This provides me with an SWF object, wrapped in JS, that performs the actual upload.

This part is working fine! Hooray!

The Problem

The problem is in tying that data back to my Django model in a sane way. Right now the data comes back as a simple URL string, pointing to the file's location.

However, I was previously using S3 Boto from django-storages to manage all of my files as FileFields, backed by the delightful S3BotoStorageFile.

To reiterate, S3 Boto is working great in isolation, Uploadify is working great in isolation, the problem is in putting the two together.

My understanding is that the only way to populate the FileField is by providing both the filename AND the file content. When you're uploading files from the browser to Django, this is no problem, as Django has the file content in a buffer and can do whatever it likes with it. However, when doing direct-to-S3 uploads like me, Django only receives the file name and URL, not the binary data, so I can't properly populate the FieldFile.

Cry For Help

Anyone know a graceful way to use S3Boto's FileField in conjunction with direct-to-S3 uploading?

Else, what's the best way to manage an S3 file just based on its URL? Including setting expiration, key id, etc.

Many thanks!

Appling

So change it to a URLField? After all you don't have the file, just a url. – Watcher

Good solution, but is there an easy way to build some S3 intelligence (expiration, credentials) into the URLField? Or do I write that logic myself? – Appling

Dig around, perhaps someone has solved this already and put up their solution; but very likely something to do yourself. – Watcher

I know this is quite old, but I am exactly at the same situation... Did you find a good solution to this problem you could share? – Hound

Answer 1

Use a URLField.

Watcher

Answer 2

I had a similar issue where i want to store file to s3 either directly using FileField or i have an option for the user to input the url directly. So to circumvent that, i used 2 fields in my model, one for FileField and one for URLField. And in the template i could use 'or' to see which one exists and to use that like {{ instance.filefield or instance.url }}.

Adp

Answer 3

This is untested, but you should be able to use:

from django.core.files.storage import default_storage
f = default_storage.open('name_you_expect_in_s3', 'r')
#f is an instance of S3BotoStorageFile, and can be assigned to a field
obj, created = YourObject.objects.get_or_create(**stuff_you_know)
obj.s3file_field = f
obj.save()

I think this should set up the local pointer to s3 and save it, without over writing the content.

ETA: You should do this only after the upload completes on S3 and you know the key in s3.

Dykstra

Answer 4

Checkout django-filetransfers. Looks like it plays nice with django-storages.

Rodger

Answer 5

I've never used django, so ymmv :) but why not just write a single byte to populate the content? That way, you can still use FieldFile.

Gabo

Because doing so actually creates a 1-byte file on disk and auto-updates the corresponding URL to point at that 1-byte file. :( – Appling

Answer 6

I'm thinking that writing actual SQL may be the easiest solution here. Alternatively you could subclass S3BotoStorage, override the _save method and allow for an optional kwarg of filepath which sidesteps all the other saving stuff and just returns the cleaned_name.

Seraph