Distributed lock manager for Python

Question

I have a bunch of servers with multiple instances accessing a resource that has a hard limit on requests per second.

I need a mechanism to lock the access on this resource for all servers and instances that are running.

There is a restful distributed lock manager I found on github: https://github.com/thefab/restful-distributed-lock-manager

Unfortunately there seems to be a min. lock time of 1 second and it's relatively unreliable. In several tests it took between 1 and 3 seconds to unlock a 1 second lock.

Is there something well tested with a python interface I can use for this purpose?

Edit: I need something that auto unlocks in under 1 second. The lock will never be released in my code.

Roadside

If you have a redis server accessible from all your machines, you could try: chris-lamb.co.uk/posts/distributing-locking-python-and-redis (there's a python package for it, and all: pypi.python.org/pypi/python-redis-lock ) – Yellowstone

looks good, but also has a min locktime of 1 second. – Roadside

Auto-unlocking seems error-prone to me. Why would you not release the lock when you are finished accessing the shared resource? – Cyprio

Answer 1

My first idea was using Redis. But there are more great tools and some are even lighter, so my solution builds on zmq. For this reason you do not have to run Redis, it is enough to run small Python script.

Requirements Review

Let me review your requirements before describing solution.

• limit number of request to some resource to a number of requests within fixed period of time.

• auto unlocking

• resource (auto) unlocking shall happen in time shorter than 1 second.

• it shall be distributed. I will assume, you mean that multiple distributed servers consuming some resource shall be able and it is fine to have just one locker service (more on it at Conclusions)

Concept

Limit number of requests within timeslot

Timeslot can be a second, more seconds, or shorter time. The only limitation is precision of time measurement in Python.

If your resource has hard limit defined per second, you shall use timeslot 1.0

Monitoring number of requests per timeslot until next one starts

With first request for accessing your resource, set up start time for next timeslot and initialize request counter.

With each request, increase request counter (for current time slot) and allow the request unless you have reached max number of allowed requests in current time slot.

Serve using zmq with REQ/REP

Your consuming servers could be spread across more computers. To provide access to LockerServer, you will use zmq.

Sample code

zmqlocker.py:

import time
import zmq

class Locker():
    def __init__(self, max_requests=1, in_seconds=1.0):
        self.max_requests = max_requests
        self.in_seconds = in_seconds
        self.requests = 0
        now = time.time()
        self.next_slot = now + in_seconds

    def __iter__(self):
        return self

    def next(self):
        now = time.time()
        if now > self.next_slot:
            self.requests = 0
            self.next_slot = now + self.in_seconds
        if self.requests < self.max_requests:
            self.requests += 1
            return "go"
        else:
            return "sorry"


class LockerServer():
    def __init__(self, max_requests=1, in_seconds=1.0, url="tcp://*:7777"):
        locker=Locker(max_requests, in_seconds)
        cnt = zmq.Context()
        sck = cnt.socket(zmq.REP)
        sck.bind(url)
        while True:
            msg = sck.recv()
            sck.send(locker.next())

class LockerClient():
    def __init__(self, url="tcp://localhost:7777"):
        cnt = zmq.Context()
        self.sck = cnt.socket(zmq.REQ)
        self.sck.connect(url)
    def next(self):
        self.sck.send("let me go")
        return self.sck.recv()

Run your server:

run_server.py:

from zmqlocker import LockerServer

svr = LockerServer(max_requests=5, in_seconds=0.8)

From command line:

$ python run_server.py

This will start serving locker service on default port 7777 on localhost.

Run your clients

run_client.py:

from zmqlocker import LockerClient
import time

locker_cli = LockerClient()

for i in xrange(100):
    print time.time(), locker_cli.next()
    time.sleep(0.1)

From command line:

$ python run_client.py

You shall see "go", "go", "sorry"... responses printed.

Try running more clients.

A bit of stress testing

You may start clients first and server later on. Clients will block until the server is up, and then will happily run.

Conclusions

• described requirements are fulfilled
  • number of requests is limited
  • no need to unlock, it allows more requests as soon as there is next time slot available
  • LockerService is available over network or local sockets.
• it shall be reliable, zmq is mature solution, python code is rather simple
• it does not require time synchronization across all participants
• performance will be very good

On the other hand, you may find, that limits of your resource are not so predictable as you assume, so be prepared to play with parameters to find proper balance and be always prepared for exceptions from this side.

There is also some space for optimization of providing "locks" - e.g. if locker runs out of allowed requests, but current timeslot is already almost completed, you might consider waiting a bit with your "sorry" and after a fraction of second provide "go".

Extending it to real distributed lock manager

By "distributed" we might also understand multiple locker servers running together. This is more difficult to do, but is also possible. zmq allows very easy connection to multiple urls, so clients could really easily connect to multiple locker servers. There is a question, how to coordinate locker servers not to allow too many request to your resource. zmq allows inter-server communication. One model could be, that each locker server would publish each provided "go" on PUB/SUB. All other locker servers would be subscribed, and used each "go" to increase their local request counter (with a bit modified logic).

Demagnetize

Answer 2

The lowest effort way to implement this is to use lockable.

It offers low-level lock semantics and it comes with a Python client. Iportantly, you don't need to set up any database or server, it works by storing the lock on the lockable servers.

Locks have variable TTLs, but you can also release them early:

$ pip install lockable-dev

from lockable import Lock

my_lock = Lock('my-lock-name')

# acquire the lock
my_lock.acquire()

# release the lock
my_lock.release()

Dipietro

The site is offline? I wonder what happened to all the apps using it. – Heartsome

Answer 3

For my cluster I'm using ZooKeeper with python-kazoo library for queues and locks.

Modified example from kazoo api documentation for your purpose: http://kazoo.readthedocs.org/en/latest/api/recipe/lock.html

zk = KazooClient()
lock = zk.Lock("/lockpath", "my-identifier")
if lock.acquire(timeout=1):

   code here

   lock.release()

But you need at least three nodes for ZooKeeper as I remember.

Faubert

Does your solution control request rate? – Demagnetize

@JanVlcinsky no, this code doesn't control request rate. For controlling request rate it will be better to put all code with locks to class method and wrap it with decorator. Decorator will storing a number of request to zookeeper node and release lock when limit is reached. – Faubert

Answer 4

Alternatively, a poor man solution is to just use filelock on top of a distributed file system! E.g. nfs, s3fs or whatnot.

import filelock

lock = filelock.FileLock("/path/to/mounted/dfs/my-id")
with lock:
    pass  # do your thing

Be cautious with your choice: the consistency of such lock will be in pair with the distributed file system's specific consistency.

Hardly

Answer 5

Alternatively, AWS DynamoDB can be used to implement a locking protocol and a libre implementation exists for python

import boto3
from python_dynamodb_lock.python_dynamodb_lock import DynamoDBLockClient


dynamodb = boto3.resource("dynamodb")

lock_client = DynamoDBLockClient(dynamodb_resource)

with lock_client.acquire_lock("my-key"):
    pass  # ... app logic that requires the lock ...

The project is sort of stale and very tailored for an AWS environment, but I think it is worth being mentioned in this question.

See

• https://github.com/mohankishore/python_dynamodb_lock
• https://aws.amazon.com/blogs/database/building-distributed-locks-with-the-dynamodb-lock-client/

Hardly

Answer 6

Your requirements seem very specific. I'd consider writing a simple lock server then implementing the locks client side with a class that acquires a lock when it is created then deletes the lock when it goes out of scope.

class Lock(object):
    def __init__(self,resource):
        print "Lock acquired for",resource
        # Connect to lock server and acquire resource

    def __del__(self):
        print "Lock released"
        # Connect to lock server and unlock resource if locked

def callWithLock(resource,call,*args,**kwargs):
    lock = Lock(resource)
    return call( *args, **kwargs )

def test( asdf, something="Else" ):
    return asdf + " " + something

if __name__ == "__main__":
    import sys
    print "Calling test:",callWithLock( "resource.test", test, sys.argv[0] )

Sample output

$ python locktest.py 
Calling test: Lock acquired for resource.test
Lock released
locktest.py Else

Benzophenone

Using __del__ to release your lock is a very bad idea. You have no guarantees how quickly Python will call that method after an object goes out of scope, especially if it is part of a reference cycle. Instead, you should use the context manager protocol (__enter__ and __exit__) and a with statement. Furthermore, I'm not sure how useful this answer is without any attempt at actually implementing the lock server or the client code you've glossed over with your # Connect to lock server comments. I suspect Jay knows how to use a lock server, but he doesn't have one! – Bowen

Well I doubt he's going to get someone to write one for him. My solution is that he roll his own. Your suggestions for using the context manager proto are spot on though, thanks! :) – Benzophenone

Answer 7

The distributed lock manager Taooka http://taooka.com has a TTL accuracy to nanoseconds. But it only has Golang client library.

Doughy