How to tell Django, that memcached running with item-size larger than default?

Question

Im using new setting to increase item size in memcached, but i cant store something larger than 1mb through Django backend. I know that memcache module require some setting to achieve thath, and Django use this module in backend.

Si

Answer 1

From Maximum size of object that can be saved in memcached with memcache.py:

There are two entries about that in the memcached FAQ :

• What is the maximum data size you can store? Why are items limited to 1 megabyte in size? The answer to the first one is (quoting, emphasis mine) :

• The maximum size of a value you can store in memcached is 1 megabyte. If your data is larger, consider clientside compression or splitting the value up into multiple keys.

So I'm guessing your 11MB file is quite too big to fit in one memcached entry.

If you do really want to cache larger objects, you will have to subclass Django's MemcachedCache as it doesn't allow you to pass in options:

self._client = self._lib.Client(self._servers, pickleProtocol=pickle.HIGHEST_PROTOCOL)

Example subclass implementation:

from django.core.cache.backends.memcached import MemcachedCache

class LargeMemcachedCache(MemcachedCache):
    "Memcached cache for large objects"

    @property
    def _cache(self):
        if getattr(self, '_client', None) is None:
            self._client = self._lib.Client(self._servers, 
                           pickleProtocol=pickle.HIGHEST_PROTOCOL, 
                           server_max_value_length = 1024*1024*10)
        return self._client

California

Answer 2

In more recent Django versions you don't need to subclass the cache class, you can instead specify the arguments passed to the cache class's constructor in OPTIONS:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
        'KEY_FUNCTION': 'some.path.to.a.function',
        'OPTIONS': {
            'server_max_value_length': 1024 * 1024 * 10
        }
    }
}

See docs.

Note that you must also increase the size in memcached itself by adding the following line:

-I 10m

to /etc/memcached.conf and restarting it:

sudo service memcached restart

Distributary

Answer 3

Django deprecated the use of Memcached after 4.1

Instead of MemcachedCache you can use PyMemcacheCache so BACKEND becomes

"BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",

like before, the default object_size_limit of memcache is 1mb, but 'serve_max_value_length' is no longer used. Whatever size limit you set manually in the memcache config will be used by PyMemcacheCache in my case, running on Debian, I had to add the following line to /etc/memcached.conf

-I 10m

this made the size limit to 10 mb

Fylfot

in my case even using -2m did the job, so be sure to experiment and decide the best size for your own specific use case – Fylfot