pylibmc: 'Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key"'

Question

I have a python web app that uses the pylibmc module to connect to a memcached server. If I test my app with requests once per second or slower, everything works fine. If I send more than one request per second, however, my app crashes and I see the following in my logs:

Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key" likely for "Programmer error, the query_id was not incremented.", at libmemcached/get.cc:107

Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key" likely for "Programmer error, the query_id was not incremented.", at libmemcached/get.cc:89

Any idea what's going wrong or how to fix it?

My code looks like this:

self.mc = pylibmc.Client(
    servers=[os.environ.get(MEMCACHE_SERVER_VAR)],
    username=os.environ.get(MEMCACHE_USER_VAR),
    password=os.environ.get(MEMCACHE_PASS_VAR),
    binary=True
    )

#...

if (self.mc != None):
    self.mc.set(key, stored_data)

#...

page = self.mc.get(key)

Bedight

Why did you immediately post an answer to your own question? – Beaker

@Beaker Because I solved the issue? AFAIK answering your own question is encouraged. Are you saying I should not? I'm open to other suggestions if other people have them, but my answer solved my issue. When I originally searched for answers to this problem I found none, so I'm hoping that posting it here helps others who run into it. – Bedight

Answer 1

This is a threading issue. pylibmc clients are not thread-safe. You should convert your code to use a ThreadMappedPool object to ensure you keep a separate connection for each thread. Something like this:

mc = pylibmc.Client(
    servers=[os.environ.get(MEMCACHE_SERVER_VAR)],
    username=os.environ.get(MEMCACHE_USER_VAR),
    password=os.environ.get(MEMCACHE_PASS_VAR),
    binary=True
    )
self.pool = pylibmc.ThreadMappedPool(mc)

#...

if (self.pool != None):
    with self.pool.reserve() as mc:
        mc.set(key, stored_data)

#...

if (self.pool != None):
    with self.pool.reserve() as mc:
        page = mc.get(key)

Make sure to call self.pool.relinquish() when the thread is finished, possibly in the destructor!

(In my case this happened because I was using cherrypy as my web server, and cherrypy spawns 10 separate threads to serve requests by default.)

Bedight

Answer 2

I ran into the same issue running Django on Apache. Switching from pylibmc to python-memcached eliminated the problem for me.

Aflutter

Also, see the django_elasticache.memcached.ElastiCache backend. It uses pylibmc, but calls it in such as way so it plays nicely with the GIL and doesn't crash. – Botts