Expire a view-cache in Django?

Question

The @cache_page decorator is awesome. But for my blog I would like to keep a page in cache until someone comments on a post. This sounds like a great idea as people rarely comment so keeping the pages in memcached while nobody comments would be great. I'm thinking that someone must have had this problem before? And this is different than caching per url.

So a solution I'm thinking of is:

@cache_page( 60 * 15, "blog" );
def blog( request ) ...

And then I'd keep a list of all cache keys used for the blog view and then have way of expire the "blog" cache space. But I'm not super experienced with Django so I'm wondering if someone knows a better way of doing this?

Hibben

Answer 1

This solution works for django versions before 1.7

Here's a solution I wrote to do just what you're talking about on some of my own projects:

def expire_view_cache(view_name, args=[], namespace=None, key_prefix=None):
    """
    This function allows you to invalidate any view-level cache. 
        view_name: view function you wish to invalidate or it's named url pattern
        args: any arguments passed to the view function
        namepace: optioal, if an application namespace is needed
        key prefix: for the @cache_page decorator for the function (if any)
    """
    from django.core.urlresolvers import reverse
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key
    from django.core.cache import cache
    # create a fake request object
    request = HttpRequest()
    # Loookup the request path:
    if namespace:
        view_name = namespace + ":" + view_name
    request.path = reverse(view_name, args=args)
    # get cache key, expire if the cached item exists:
    key = get_cache_key(request, key_prefix=key_prefix)
    if key:
        if cache.get(key):
            # Delete the cache entry.  
            #
            # Note that there is a possible race condition here, as another 
            # process / thread may have refreshed the cache between
            # the call to cache.get() above, and the cache.set(key, None) 
            # below.  This may lead to unexpected performance problems under 
            # severe load.
            cache.set(key, None, 0)
        return True
    return False

Django keys these caches of the view request, so what this does is creates a fake request object for the cached view, uses that to fetch the cache key, then expires it.

To use it in the way you're talking about, try something like:

from django.db.models.signals import post_save
from blog.models import Entry

def invalidate_blog_index(sender, **kwargs):
    expire_view_cache("blog")

post_save.connect(invalidate_portfolio_index, sender=Entry)

So basically, when ever a blog Entry object is saved, invalidate_blog_index is called and the cached view is expired. NB: haven't tested this extensively, but it's worked fine for me so far.

Cedar

It does the trick really well with database cache, but doesn't work for me with filesystem cache: key = get_cache_key(request) returns None with filesystem cache, but returns the correct key with database cache. Have you got an idea on how to make it run with filesystem cache backend? – Tananarive

get_cache_key returns nothing. – Opinionated

after you unset the cache, would django automatically cache_page it again next time? It doesn't seem to work at least with redis cache backend. – Returnable

I think args=[] in argument list is dangerous isn't it? Try this def my_func(args=[]): args.append('a') return args for i in xrange(5): print my_func() – Kampmeier

try but does not work for me. This line: key = get_cache_key(request, key_prefix=key_prefix) return key = None, meaning request is not found in cache, but in fact, it is there in the cache. – Chandlery

Does not work for me. I think Django has successfully locked down per-view cache so it cannot be invalidated before it times out. – Melquist

Django use the meta data of the request to build the key in cache. If the initial request are made in a different language / different timezone, the key won't match in the cache and you will end up with key = None. One other common issue is the request.META["SERVER_NAME"] being different (eg: 127.0.0.1:8000 vs localhost:80) – Touchline

this is my solution: https://mcmap.net/q/330485/-clearing-specific-cache-in-django – Nodab

Answer 2

The cache_page decorator will use CacheMiddleware in the end which will generate a cache key based on the request (look at django.utils.cache.get_cache_key) and the key_prefix ("blog" in your case). Note that "blog" is only a prefix, not the whole cache key.

You can get notified via django's post_save signal when a comment is saved, then you can try to build the cache key for the appropriate page(s) and finally say cache.delete(key).

However this requires the cache_key, which is constructed with the request for the previously cached view. This request object is not available when a comment is saved. You could construct the cache key without the proper request object, but this construction happens in a function marked as private (_generate_cache_header_key), so you are not supposed to use this function directly. However, you could build an object that has a path attribute that is the same as for the original cached view and Django wouldn't notice, but I don't recommend that.

The cache_page decorator abstracts caching quite a bit for you and makes it hard to delete a certain cache object directly. You could make up your own keys and handle them in the same way, but this requires some more programming and is not as abstract as the cache_page decorator.

You will also have to delete multiple cache objects when your comments are displayed in multiple views (i.e. index page with comment counts and individual blog entry pages).

To sum up: Django does time based expiration of cache keys for you, but custom deletion of cache keys at the right time is more tricky.

Immingle

Answer 3

I wrote Django-groupcache for this kind of situations (you can download the code here). In your case, you could write:

from groupcache.decorators import cache_tagged_page

@cache_tagged_page("blog", 60 * 15)
def blog(request):
    ...

From there, you could simply do later on:

from groupcache.utils import uncache_from_tag

# Uncache all view responses tagged as "blog"
uncache_from_tag("blog") 

Have a look at cache_page_against_model() as well: it's slightly more involved, but it will allow you to uncache responses automatically based on model entity changes.

Aphonic

@laike9m latest commit was in 2013. I am going to say it's a dead project now however that doesn't mean it won't still work with latest versions of Django. – Logo

well, the repository is unreachable. i think we can assume it is dead. – Zubkoff

Answer 4

This won't work on django 1.7; as you can see here https://docs.djangoproject.com/en/dev/releases/1.7/#cache-keys-are-now-generated-from-the-request-s-absolute-url the new cache keys are generated with the full URL, so a path-only fake request won't work. You must setup properly request host value.

fake_meta = {'HTTP_HOST':'myhost',}
request.META = fake_meta

If you have multiple domains working with the same views, you should cycle them in the HTTP_HOST, get proper key and do the clean for each one.

Extremism

Appart from the HTTP_HOST you need the SERVER_PORT too, but I could not be able to retrieve the right key with this method (in devel enviroment): fake_meta = {'HTTP_HOST':'127.0.0.1', 'SERVER_PORT': 8000} request.META = fake_meta – Menke

Answer 5

With the latest version of Django(>=2.0) what you are looking for is very easy to implement:

from django.utils.cache import learn_cache_key
from django.core.cache import cache
from django.views.decorators.cache import cache_page

keys = set()

@cache_page( 60 * 15, "blog" );
def blog( request ):
    response = render(request, 'template')
    keys.add(learn_cache_key(request, response)
    return response

def invalidate_cache()
    cache.delete_many(keys)

You can register the invalidate_cache as a callback when someone updates a post in the blog via a pre_save signal.

Shimmery

actually this one doesn't work correctly, learn_cache_key returns different key when you generate it yourself in the code, and in the cache middleware.. – Returnable

Also: print(keys) executed before and after cache.delete_many(keys) gives the same output. Something else maybe is needed, like some kind of .save() ? – Cowardly

keys is local to the thread. This won't work when the app runs in multiple threads/processes – as it likely will in a production environment. – Immingle

Hey, wouldn't this give problems in the case of Gunicorn or UWSGI where we have multiple processes? Each process starts a new program and thus has its own keys. However, a request might arrive on different processes? – Xenogamy

I am still being served the cache data from disk even tough my redis keys are empty – Furnishings

Answer 6

Django view cache invalidation for v1.7 and above. Tested on Django 1.9.

def invalidate_cache(path=''):
    ''' this function uses Django's caching function get_cache_key(). Since 1.7, 
        Django has used more variables from the request object (scheme, host, 
        path, and query string) in order to create the MD5 hashed part of the
        cache_key. Additionally, Django will use your server's timezone and 
        language as properties as well. If internationalization is important to
        your application, you will most likely need to adapt this function to
        handle that appropriately.
    '''
    from django.core.cache import cache
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key

    # Bootstrap request:
    #   request.path should point to the view endpoint you want to invalidate
    #   request.META must include the correct SERVER_NAME and SERVER_PORT as django uses these in order
    #   to build a MD5 hashed value for the cache_key. Similarly, we need to artificially set the 
    #   language code on the request to 'en-us' to match the initial creation of the cache_key. 
    #   YMMV regarding the language code.        
    request = HttpRequest()
    request.META = {'SERVER_NAME':'localhost','SERVER_PORT':8000}
    request.LANGUAGE_CODE = 'en-us'
    request.path = path

    try:
        cache_key = get_cache_key(request)
        if cache_key :
            if cache.has_key(cache_key):
                cache.delete(cache_key)
                return (True, 'successfully invalidated')
            else:
                return (False, 'cache_key does not exist in cache')
        else:
            raise ValueError('failed to create cache_key')
    except (ValueError, Exception) as e:            
        return (False, e)

Usage:

status, message = invalidate_cache(path='/api/v1/blog/')

Mancunian

Answer 7

I had same problem and I didn't want to mess with HTTP_HOST, so I created my own cache_page decorator:

from django.core.cache import cache


def simple_cache_page(cache_timeout):
    """
    Decorator for views that tries getting the page from the cache and
    populates the cache if the page isn't in the cache yet.

    The cache is keyed by view name and arguments.
    """
    def _dec(func):
        def _new_func(*args, **kwargs):
            key = func.__name__
            if kwargs:
                key += ':' + ':'.join([kwargs[key] for key in kwargs])

            response = cache.get(key)
            if not response:
                response = func(*args, **kwargs)
                cache.set(key, response, cache_timeout)
            return response
        return _new_func
    return _dec

To expired page cache just need to call:

cache.set('map_view:' + self.slug, None, 0)

where self.slug - param from urls.py

url(r'^map/(?P<slug>.+)$', simple_cache_page(60 * 60 * 24)(map_view), name='map'), 

Django 1.11, Python 3.4.3

Keeton

I'm a beginner. Do you have any working example on github? – Karaganda

Answer 8

FWIW I had to modify mazelife's solution to get it working:

def expire_view_cache(view_name, args=[], namespace=None, key_prefix=None, method="GET"):
    """
    This function allows you to invalidate any view-level cache. 
        view_name: view function you wish to invalidate or it's named url pattern
        args: any arguments passed to the view function
        namepace: optioal, if an application namespace is needed
        key prefix: for the @cache_page decorator for the function (if any)

        from: https://mcmap.net/q/326290/-expire-a-view-cache-in-django
        added: method to request to get the key generating properly
    """
    from django.core.urlresolvers import reverse
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key
    from django.core.cache import cache
    # create a fake request object
    request = HttpRequest()
    request.method = method
    # Loookup the request path:
    if namespace:
        view_name = namespace + ":" + view_name
    request.path = reverse(view_name, args=args)
    # get cache key, expire if the cached item exists:
    key = get_cache_key(request, key_prefix=key_prefix)
    if key:
        if cache.get(key):
            cache.set(key, None, 0)
        return True
    return False

Seizing

Answer 9

Instead of using the cache page decorator, you could manually cache the blog post object (or similar) if there are no comments, and then when there's a first comment, re-cache the blog post object so that it's up to date (assuming the object has attributes that reference any comments), but then just let that cached data for the commented blog post expire and then no bother re-cacheing...

Ugly

I think caching the whole response might be an idea. Looking into that. Thanks. – Hibben

Answer 10

Instead of explicit cache expiration you could probably use new "key_prefix" every time somebody comment the post. E.g. it might be datetime of the last post's comment (you could even combine this value with the Last-Modified header).

Unfortunately Django (including cache_page()) does not support dynamic "key_prefix"es (checked on Django 1.9) but there is workaround exists. You can implement your own cache_page() which may use extended CacheMiddleware with dynamic "key_prefix" support included. For example:

from django.middleware.cache import CacheMiddleware
from django.utils.decorators import decorator_from_middleware_with_args

def extended_cache_page(cache_timeout, key_prefix=None, cache=None):
    return decorator_from_middleware_with_args(ExtendedCacheMiddleware)(
        cache_timeout=cache_timeout,
        cache_alias=cache,
        key_prefix=key_prefix,
    )

class ExtendedCacheMiddleware(CacheMiddleware):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if callable(self.key_prefix):
            self.key_function = self.key_prefix

    def key_function(self, request, *args, **kwargs):
        return self.key_prefix

    def get_key_prefix(self, request):
        return self.key_function(
            request,
            *request.resolver_match.args,
            **request.resolver_match.kwargs
        )

    def process_request(self, request):
        self.key_prefix = self.get_key_prefix(request)
        return super().process_request(request)

    def process_response(self, request, response):
        self.key_prefix = self.get_key_prefix(request)
        return super().process_response(request, response)

Then in your code:

from django.utils.lru_cache import lru_cache

@lru_cache()
def last_modified(request, blog_id):
    """return fresh key_prefix"""

@extended_cache_page(60 * 15, key_prefix=last_modified)
def view_blog(request, blog_id):
    """view blog page with comments"""

Hushaby

Answer 11

Most of the solutions above didn't work in our case because we use https. The source code for get_cache_key reveals that it uses request.get_absolute_uri() to generate the cache key.

The default HttpRequest class sets the scheme as http. Thus we need to override it to use https for our dummy request object.

This is the code that works fine for us :)

from django.core.cache import cache
from django.http import HttpRequest
from django.utils.cache import get_cache_key


class HttpsRequest(HttpRequest):
    @property
    def scheme(self):
        return "https"


def invalidate_cache_page(
    path,
    query_params=None,
    method="GET",
):
    request = HttpsRequest()

    # meta information can be checked from error logs
    request.META = {
        "SERVER_NAME": "www.yourwebsite.com",
        "SERVER_PORT": "443",
        "QUERY_STRING": query_params,
    }

    request.path = path
    key = get_cache_key(request, method=method)
    if cache.has_key(key):
        cache.delete(key)

Now I can use this utility function to invalidate the cache from any of our views:

page = reverse('url_name', kwargs={'id': obj.id})
invalidate_cache_page(path)

Xenogamy

Answer 12

Duncan's answer works well with Django 1.9. But if we need invalidate url with GET-parameter we have to make a little changes in request. Eg for .../?mykey=myvalue

request.META = {'SERVER_NAME':'127.0.0.1','SERVER_PORT':8000, 'REQUEST_METHOD':'GET', 'QUERY_STRING': 'mykey=myvalue'}
request.GET.__setitem__(key='mykey', value='myvalue')

Reportorial

Answer 13

I struggled with a similar situation and here is the solution I came up with, I started it on an earlier version of Django but it is currently in use on version 2.0.3.

First issue: when you set things to be cached in Django, it sets headers so that downstream caches -- including the browser cache -- cache your page.

To override that, you need to set middleware. I cribbed this from elsewhere on StackOverflow, but can't find it at the moment. In appname/middleware.py:

from django.utils.cache import add_never_cache_headers


class Disable(object):

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        add_never_cache_headers(response)
        return response

Then in settings.py, to MIDDLEWARE, add:

'appname.middleware.downstream_caching.Disable',

Keep in mind that this approach completely disables downstream caching, which may not be what you want.

Finally, I added to my views.py:

def expire_page(request, path=None, query_string=None, method='GET'):
    """
    :param request: "real" request, or at least one providing the same scheme, host, and port as what you want to expire
    :param path: The path you want to expire, if not the path on the request
    :param query_string: The query string you want to expire, as opposed to the path on the request
    :param method: the HTTP method for the page, if not GET
    :return: None
    """
    if query_string is not None:
        request.META['QUERY_STRING'] = query_string
    if path is not None:
        request.path = path
    request.method = method

    # get_raw_uri and method show, as of this writing, everything used in the cache key
    # print('req uri: {} method: {}'.format(request.get_raw_uri(), request.method))
    key = get_cache_key(request)
    if key in cache:
        cache.delete(key)

I didn't like having to pass in a request object, but as of this writing, it provides the scheme/protocol, host, and port for the request, pretty much any request object for your site/app will do, as long as you pass in the path and query string.

Edmond

Answer 14

One more updated version of Duncan's answer: had to figure out correct meta fields: (tested on Django 1.9.8)

def invalidate_cache(path=''):
    import socket
    from django.core.cache import cache
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key

    request = HttpRequest()
    domain = 'www.yourdomain.com'
    request.META = {'SERVER_NAME': socket.gethostname(), 'SERVER_PORT':8000, "HTTP_HOST": domain, 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, br'}
    request.LANGUAGE_CODE = 'en-us'
    request.path = path

    try:
        cache_key = get_cache_key(request)
        if cache_key :
            if cache.has_key(cache_key):
                cache.delete(cache_key)
                return (True, 'successfully invalidated')
            else:
                return (False, 'cache_key does not exist in cache')
        else:
            raise ValueError('failed to create cache_key')
    except (ValueError, Exception) as e:            
        return (False, e)

Corposant

Answer 15

A simple helper function that clears the cache for a given URL, assuming no headers. Probably best used when wired up to a model's post_save event via signals. See message to django-users mailing list for background.

from django.core.cache import cache
from django.http import HttpRequest
from django.utils.cache import get_cache_key

def expire_page(path):
    request = HttpRequest()
    request.path = path
    key = get_cache_key(request)
    if cache.has_key(key):   
        cache.delete(key)

Beall

Answer 16

The solution is simple, and do not require any additional work.

Example

@cache_page(60 * 10)
def our_team(request, sorting=None):
    ...

This will set the response to the cache with the default key.

Expire a view cache

from django.utils.cache import get_cache_key
from django.core.cache import cache

def our_team(request, sorting=None):
    # This will remove the cache value and set it to None
    cache.set(get_cache_key(request), None)

Simple, Clean, Fast.

Meander