Synchronising multiple threads in python

Question

I have a problem where I need x threads to wait until they have all reached a synchronization point. My solution uses the synchronise method below which is called by each threaded function when they need to synchronise.

Is there a better way to do this?

thread_count = 0
semaphore = threading.Semaphore()
event = threading.Event()

def synchronise(count):
    """ All calls to this method will block until the last (count) call is made """
    with semaphore:
        thread_count += 1
        if thread_count == count:
            event.set()

    event.wait()

def threaded_function():
    # Do something

    # Block until 4 threads have reached this point
    synchronise(4)

    # Continue doing something else

Trinitrocresol

Answer 1

There are many ways to synchronize threads. Many.

In addition to synchronize, you can do things like the following.

1. Break your tasks into two steps around the synchronization point. Start threads doing the pre-sync step. Then use "join" to wait until all threads finish step 1. Start new threads doing the post-sync step. I prefer this, over synchronize.

2. Create a queue; acquire a synchronization lock. Start all threads. Each thread puts an entry in the queue and waits on the synchronization lock. The "main" thread sits in a loop dequeueing items from the queue. When all threads have put an item in the queue, the "main" thread releases the synchronization lock. All other threads are now free to run again.

There are a number of interprocess communication (IPC) techniques -- all of which can be used for thread synchronization.

Plotkin

I explored your first suggestion but there was a need to have the threads doing both the before synchronisation and after synchronisation work without splitting the effort into 2 tasks. If I didn't have this constraint your solution would be be ideal. – Trinitrocresol

Since threads all share the exact same memory, It's not clear why your threads can't be decomposed into two steps. The information created pre-synch should be completely available for new threads to use post-synch. – Plotkin

Answer 2

Note that Barrier has been implemented as of Python 3.2

Example of using barriers:

from threading import Barrier, Thread

def get_votes(site):
    ballots = conduct_election(site)
    all_polls_closed.wait()        # do not count until all polls are closed
    totals = summarize(ballots)
    publish(site, totals)

all_polls_closed = Barrier(len(sites))
for site in sites:
    Thread(target=get_votes, args=(site,)).start()

Tergum

Answer 3

The functionality you want is called a "barrier". (Unfortunately that term has 2 meanings when talking about threading. So if you Google it, just ignore articles that talk about "memory barriers" - that's a very different thing).

Your code looks quite reasonable - it's simple and safe.

I couldn't find any "standard" implementations of barriers for Python, so I suggest you keep using your code.

Abradant