How do I use Django, sorl-thumbnail, memcached, and S3 together?

Question

I've got a project in which I need to start creating dynamically-resized thumbnails of user-uploaded images, where previously I'd been generating some specifically sized ones. I have an existing model, which represents an image which looks like this:

class Image(models.Model):
  original = models.URLField()
  small = models.URLField()
  medium = models.URLField()
  large = models.URLField()

Each of those is filled in with a URL to the respective image on S3. At the time of upload, all the versions are created (actually shortly after, via a scheduled task), stored in S3, and then set on the model. Going forward, our design needs many different sizes for each thumbnail, so storing them all in the db is not practical. I'd like to just keep original and do the rest on-the-fly in the templates.

So this is where I come to sorl-thumbnail. It seems to be intended to do what I want, but I'm kind of lost as to how it's supposed to be working (it's not at the moment). As a test I've taken made a basic view/template which lists some images and attempts to thumbnail them, much like the tutorial:

{% for img in image_set %}
  {% thumbnail img.original "180x180" crop="center" format="PNG" as im %}
  <img src="{{ im.url }}" width="{{ im.width }}" height="{{ im.height }}">
  {% endthumbnail %}
{% endfor %}

That ends up generating a URL for the file like cache/84/8f/848fb078f2e8d35aecd92150c6fa6843.png (which 404s). I'm not sure I understand where this file is supposed to be getting stored. I am assuming it is pulling in the S3 file and then creating the thumbnail somewhere, but I am unclear where that is, and where I am supposed to be serving it from. This is all on the local devserver right now, using memcached (in which I can see is getting cache entries set). PIL is installed in my environment with jpeg/png support.

Payola

This gives a schematic idea of how it works docs.google.com/drawings/…. Your thumbnail is getting stored in memcached right? So that somewhere is memcached. If you can't access the cache it is a different issue there. – Sepal

Er, hmm yes I suppose that may be the problem. How do I access the files in memcached? It's running locally at 127.0.0.1:11211. – Payola

telnet localhost 11211; get <your key>; More commands - code.google.com/p/memcached/wiki/NewCommands#stats – Sepal

Sorry, no I mean in the browser. I don't understand how I need to get the files out of memcached to serve them. – Payola

Answer 1

You might want to look at Django-storages it was a nice AWS S3 that would fix the issue for you.Along with that also install boto, as django-storage has a dependency on boto.

Then you will have to add the following into your settings.py

import os

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = '<YOUR BUCKET NAME>'

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

Then your cache will be stored on AWS S3 itself.

I hope it helps

Note: for security reasons its a good idea to add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables instead of just writing them down in setting.py directly.

Mcilroy

Thanks very much, this worked. It was the DEFAULT_FILE_STORAGE setting I needed. I had a feeling I had all the pieces there and just wasn't putting them together properly. – Payola

Pretty sure STATICFILES_STORAGE, STATIC_URL and ADMIN_MEDIA_PREFIX is unrelated to this question? I believe sort-thumbnail only runs on the DEFAULT_FILE_STORAGE. – Subsolar