Django + S3 (boto) + Sorl Thumbnail: Suggestions for optimisation

Question

I am using S3 storage backend across a Django site I am developing, both to reduce load from the EC2 server(s), and to allow multiple webservers (redundancy, load balancing) access the same set of uploaded media.

Sorl.thumbnail (v11) template tags are being used in our templates to allow flexible image resizing/cropping.

Performance on media-rich pages is not very good, and when a page containing thumbnails needing to be generated for the first time is accessed, the requests even time out.

I understand that this is due to sorl thumbnail checking/downloading the original image from S3 (which could be quite large and high resolution), and rendering/checking/uploading the thumbnail.

What would you suggest is the best solution to this setup?

I have seen suggestions of storing a local copy of files in addition to the S3 copy (not to great when a couple of server are being used for load balancing). Also I've seen it suggested to store 0-byte files to fool sorl.thumbnail.

Are there any other suggestions or better ways of approaching this?

Halfpenny

Just a heads-up for Flask users that need thumbnails on S3. I forked flask-thumbnails and created flask-thumbnails-s3. Unfortunately, doesn't do any fancy queueing or async stuff, like the various Django-based solutions linked above (contributions welcome!)... but it works. – Disappointed

Answer 1

sorl thumbnail is now created with remote slow storages in mind. The first creation of the thumbnail is however done quering the storage, for example first accessed from template, but after that the references are cached in a key value store. Still you need the first query and creation, well one solution is to use the low level api sorl.thumbnail.get_thumbnail with the same options when the image is uploaded. When the image uploaded add this thumbnail creation job to a que like celery.

Somatology

Would this still be the best solution for the issue? – Possessive

@Somatology does sorl thumbnail still access the storage if the keystores are generated? – Valval

Answer 2

You can use Sorlery. It combines sorl and celery to create thumbnails via workers. It's very careful not to do any filesystem access outside of the worker thread.

The thumbnail returned immediately (before the worker has had a chance) can be controlled by setting your THUMBNAIL_DUMMY_SOURCE to an appropriate placeholder.

The job is created the first time the thumbnail is requested, subsequent requests are served the dummy image until the worker thread completes.

Demy

I did exactly same to work with S3, but I have one problem though. If I refresh browser, same task is repeated until thumbnail generation is completed and cached. Looks you didn't face this problem. But how would you solve it? Seems celery doesn't know same task is repeating. – Cavesson

Answer 3

Almost same as @Aidan's solution, I have made some tweaks on sorl-thumbnail. I also pre-generate thumbnails with celery. My code is here sorl_thumbnail-async

But I came to know easy_thumbnails does exactly what I was trying to do, so I am using it in my current project. You might find useful, short post on the topic is here

Cavesson

the last link is 404'd :( – Disparage

Answer 4

The easiest solution I've found so far is actually this third party service: http://cloudinary.com/

Demy

Looks like a very cool service - would make thumbnail generation for python (and other) apps much easier. Thanks for pointing it out! – Disappointed