Named semaphores in Python?

Question

I have a script in python which uses a resource which can not be used by more than a certain amount of concurrent scripts running.

Classically, this would be solved by a named semaphores but I can not find those in the documentation of the multiprocessing module or threading .

Am I missing something or are named semaphores not implemented / exposed by Python? and more importantly, if the answer is no, what is the best way to emulate one?

Thanks, Boaz

PS. For reasons which are not so relevant to this question, I can not aggregate the task to a continuously running process/daemon or work with spawned processes - both of which, it seems, would have worked with the python API.

Urbain

Answer 1

I suggest a third party extension like these, ideally the posix_ipc one -- see in particular the semaphore section in the docs.

These modules are mostly about exposing the "system V IPC" (including semaphores) in a unixy way, but at least one of them (posix_ipc specifically) is claimed to work with Cygwin on Windows (I haven't verified that claim). There are some documented limitations on FreeBSD 7.2 and Mac OSX 10.5, so take care if those platforms are important to you.

Catherin

In 2023, is the Python library's multiprocessing module adequate? – Reames

Answer 2

You can emulate them by using the filesystem instead of a kernel path (named semaphores are implemented this way on some platforms anyhow). You'll have to implement sem_[open|wait|post|unlink] yourself, but it ought to be relatively trivial to do so. Your synchronization overhead might be significant (depending on how often you have to fiddle with the semaphore in your app), so you might want to initialize a ramdisk when you launch your process in which to store named semaphores.

Alternatively if you're not comfortable rolling your own, you could probably wrap boost::interprocess::named_semaphore (docs here) in a simple extension module.

Copeck

There are plenty of ways to get this wrong (fairness, scheduling wakeup, race conditions). Use kernel or libc primitives (eg. the real sem_* API, or the CreateSemaphore/event API in windows); avoid rolling your own synchronization primitives. – Lantern

This is why I offer that you could use boost if you're not comfortable rolling your own. – Copeck

@NickBastin Is there any way at all to "roll your own"? (Short of wrapping libc or Boost?) You said "You'll have to implement sem_[open|wait|post|unlink] yourself, but it ought to be relatively trivial to do so" but, the question of (non-)triviality aside, how exactly would you do that in light of possible race conditions? How would you ensure atomicity of semaphore operations? – Myceto