Should I configure Beaker's `session.lock_dir` when using Memcache on Heroku?

Question

I am switching a Pyramid application's session configuration from cookie to ext:memcached. My application is hosted on Heroku and I have configured their memcache addon as per their documentation.

I understand from the Beaker documentation that specifying a session.lock_dir is essential to preventing the dog pile effect. To be explicit: I must provide a file path to a directory. Beaker then uses this as some kind of lock to prevent multiple clients all trying to set the same value at the same time.

To me, this sounds like bad architecture. One of the primary advantages of memcache is that it works as a shared external service. Binding my application processes to a disk bound lock seems like the wrong approach.

Equally, on Heroku, I have one ephemeral filesystem per "dyno" (which I understand to mean per process). So whilst I can provide a lock_dir directory path, if each process uses a different directory, then would this protect me from the dog pile effect?

I'm not sure whether I should:

• specify a lock_dir and not worry about it
• attempt to fork Beaker and patch this somehow

Plus I'd love to know what pattern other languages / frameworks use here. Is this just a Beaker problem or do other non-file-bound setups suffer from the dog pile effect?

Thanks in advance,

James.

Shere

I'm fairly certain that memcached's backend does not require a lock_dir, have you tried it without? – Augustin

Hi Michael, beaker.ext.memcached requires a lock_dir, when using any of the backends apart from memory or cookie. My traceback here. – Shere

Sorry, my last comment didn't actually make sense. The memcached backend does require a lock_dir. The traceback I posted is triggered by this code. I've come across this discussion of the same issue. Looks like the answer is: "specify a lock_dir and not worry about it". – Shere

possible duplicate of Pyramid with memcached: how to make it work? Error - MissingCacheParameter: url is required – Pinelli

Answer 1

Unfortunately, according to my experience, Beaker does not work well on multihost infrastructure due to absence of distributed lock. If you use beaker in web app running on 2 hosts, dog pile effect still happens (however it can be avoided by sticking all requests from the same user to one server, but it does not work on all platforms).

Sometimes you can just ignore dog pile effect, sometimes not. E.g. if you cache global data, dog pile effect is pain. If you cache user data, you can ignore it sometimes.

In your case, I would ignore it. Beakers allows you to use Memcache extension without any lock dir. In this case, Beaker locks work per process only using thread locks.

Here is proof:

from beaker.middleware import SessionMiddleware
from werkzeug.wrappers import Response
from werkzeug.serving import run_simple


def simple_app(environ, start_response):
    # Get the session object from the environ
    session = environ['beaker.session']

    # Check to see if a value is in the session
    user = 'user_id' in session

    # Set some other session variable
    session['user_id'] = 10
    session.save()

    start_response('200 OK', [('Content-type', 'text/plain')])
    return ['User is logged in: %s' % user]

# Configure the SessionMiddleware
session_opts = {
    'session.type': 'ext:memcached',
    'session.url': '127.0.0.1:11211',
    'session.auto': True,
    'session.cookie_expires': True,
}
wsgi_app = SessionMiddleware(simple_app, session_opts)


if __name__ == '__main__':
    run_simple('127.0.0.1', 5000, wsgi_app, use_debugger=True, use_reloader=True)

If dog pile effect is still a problem for you, I would suggest to migrate from Beaker to something else. E.g. dogpile.cache

Armil