How to configure Memcache for Django on Google cloud AppEngine?

Question

In my settings i have:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

and i have installed https://pypi.org/project/python-memcached/

Anything else needed?

Spitler

Answer 1

The way to enable Memcache for Django on GCP depends on which product you are using (App Engine, Compute Engine, Kubernetes Engine) [1]

Since you are using App Engine, it is built-in. You can find examples on how to use Memcache with GCP here. [2]

[1] How to install Memcache on GCP for Django: https://cloud.google.com/python/django/#caches

[2]Memcache examples on GCP Docs: https://cloud.google.com/appengine/docs/standard/python/memcache/examples?hl=en

Whitewood

Answer 2

An update for anyone who's trying to do this in 2023: the accepted answer is a good start, but is incomplete. There are a number of inconsistencies between the google.appengine.api.memcache interface and the django.core.cache.backends.memcached.BaseMemcachedCache interface that need to be resolved.

I am using Django 3.2 in a Google App Engine Python 3.10 runtime.

The most important inconsistencies for my use case were

1. GAE's memcache.get does not support BaseMemcachedCache.get's default parameter. This results in a mystifying TypeError: Cannot set google.appengine.MemcacheGetRequest.name_space error.
2. Google's memcache.incr and memcache.decr support an initial_value parameter which is not supported by BaseMemcachedCache.incr/decr.
3. An important corollary to (1) is that GAE's memcache does not support the default object pattern for disambiguating cache misses from legitimately cached None values.

The implementation below resolves these inconsistencies for my use case. I'm sure there are other inconsistencies between GAE's memcache API and BaseMemcachedCache, but they're not currently relevant to my use case. Feel free to use this as a starting point for your own implementation.

from google.appengine.api import memcache

from django.utils.functional import cached_property
from django.core.cache.backends.base import DEFAULT_TIMEOUT
from django.core.cache.backends.memcached import BaseMemcachedCache


class GaeMemcachedCache(BaseMemcachedCache):
    def __init__(self, server, params):
        super().__init__(server, params, library=memcache, value_not_found_exception=ValueError)

    def decr(self, key, delta=1, version=None, initial_value=None):
        if delta < 0:
            return self.incr(key, delta=-delta, version=version, initial_value=initial_value)

        key = self.make_key(key, version=version)
        self.validate_key(key)

        value = self._cache.decr(key, delta=delta, initial_value=initial_value)

        if value is None:
            raise ValueError(f"Key '{key}' not found")

        return value

    def get(self, key, default=None, version=None):
        if not self.has_key(key, version=version):
            return default

        key = self.make_key(key, version=version)
        self.validate_key(key)

        return self._cache.get(key)

    def has_key(self, key, version=None):
        """
        Returns `True` if a key is present in the cache, otherwise `False`.

        `google.appengine.api.memcache.get` doesn't support a `default` argument. Instead, it
        returns `None` for a cache miss. `replace()`, however, returns False if the intended
        key isn't present in the cache. We can therefore check to see if key is present by
        `get`ing it and then replacing it with itself.

        `google.appengine.api.memcache` also doesn't support a way to determine a key's
        expiration time, so this approach results in the loss of that information, but
        only for keys whose value was positively set to `None`.
        """
        key = self.make_key(key, version=version)
        self.validate_key(key)

        value = self._cache.get(key)

        if value is None:
            return self._cache.replace(key, value)

        return True

    def incr(self, key, delta=1, version=None, initial_value=None):
        if delta < 0:
            return self.decr(key, delta=-delta, version=version, initial_value=initial_value)

        key = self.make_key(key, version=version)
        self.validate_key(key)

        value = self._cache.incr(key, delta=delta, initial_value=initial_value)

        if value is None:
            raise ValueError(f"Key '{key}' not found")

        return value

    @cached_property
    def _cache(self):
        if getattr(self, '_client', None) is None:
            self._client = self._lib._CLIENT
        return self._client

Cylinder