Guaranteed memcached lock

Question

So, I'm trying to implement a distributed lock using memcached and add()'s store only if does not exist contract (Java & spymemcached, but applicable in any language of course). Of course, if an instance goes away, then we lose the lock so the thought was add the lock 3 times (e.g. MyLock1, MyLock2, MyLock3) which will very likely hash out to 3 different instances.

But, I've realized that if an instances goes down the hash then obviously changes (using spymemcached's Redistribute failure mode) and so it's likely that when another attempt is made to add() the locks, that the hashes of all 3 lock's will not match any of the 2 remaining locks in the memcached cluster.

So...any other ideas for distributed locks using memcached? Or is it basically impossible to do a guaranteed lock like what I'm referring to?

EDIT: Ok, so in looking through spymemcached source code, for Redistribute mode, it simply goes to the next active memcached instance in it's list, rather than re-hashing anything, so it should work OK.

Rumery

You should consider Membase as a solution as well. Membase is basically memcached with persistence and replication so it will actually work better than just plain memcached for this problem. – Orient

Answer 1

If you really want to use memcached to avoid introducing more stuff/complexity into your environment, then consider a very small but dedicated memcached config just for locking.

But if you're open to solutions that don't rely on memcached, then personally I'd use zookeeper to implement a distributed lock in java. I'd also use the Netflix curator utils to make this easier.

Assumptive

Personally, I'd rather use zookeeper or similar, but it's one of those things that we don't currently use, would be another thing to manage with our (very) limited resources, and if this solution is a good enough solution, than it'll work for us. – Rumery

It's been my experience with locking/multithreading that you should do it right. How much contention on these locks will there be? How often are they acquired, and how long are they held? Maybe we can suggest a lower tech solution that would be more reliable than memcached and performant enough for your purposes. – Assumptive

Low contention. It's more to keep multiple instances from downloading the same FTP file. Part of the process is to append the instance hostname to the file to prevent other threads/instances from downloading it, but one of the FTP servers that we need to talk to has a distributed FS which the rename can sometimes take seconds to propagate, and so if multiple instances are connected to multiple locations that the propagation hasn't completed yet, then it can rename/start downloading the file and we get the file processed twice, which can be a Bad Thing(tm). – Rumery

Answer 2

You can't, at least not reliably. memcached doesn't make any guarantees about data retention -- as a cache, it may discard data at any point without warning. Even if the memcache instance appears to have space available, it may have to evict data due to slab constraints.

If you need distributed locking, you'll need to look elsewhere -- memcached is the wrong tool for the job. For what it's worth, MySQL has locks: http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html

Berneta

We don't need the lock for a long time, that point was considered, and if even 1 of the 3 keys exist, that means the lock could not be acquired. – Rumery

It should be very reliable in practice as long as the data set you're caching is very small (e.g. a few locks) and you're using a dedicated cache. – Assumptive

@duskwuff does this apply to #37459087 my main doubt is would memcache.get and set always return value if it exists or would it return nothing sometimes because of performance reasons where it is not able to serve the request fast enough etc. Can it "miss". – Catheryncatheter

Answer 3

If you're using Java, I recommend using Hazelcast (1.9+), it does support distributed locks across a Cluster, and is easy to create one.

Hazelcast guarantees that if a server that holds a lock goes down, the lock will be released.

http://hazelcast.com/docs/1.9.4/manual/single_html/#Lock

Also Hazelcast exposes the same contract as a memcached, so if you need to access it out of the JVM you can do it (this sample demonstrates that any client would work):

Java:

MemcachedClient client = new MemcachedClient(AddrUtil.getAddresses("10.20.17.1:5701 10.20.17.2:5701"));
client.set("key1", 3600, "value1");
System.out.println(client.get("key1"));

PHP:

<?php
    $memcache = new Memcache;
    $memcache->connect('10.20.17.1', 5701) or die ("Could not connect");
    $memcache->set('key1','value1',0,3600);
    $get_result = $memcache->get('key1'); //retrieve your data
    var_dump($get_result); //show it
?>

Documentation of 1.9+: http://hazelcast.com/docs/1.9.4/manual/single_html/

Hope it helps.

Consolata