Multiprocessing in Python: execute two functions at the exact same time

Question

I want to execute the following two functions at exactly the same time.

from multiprocessing import Process
import os
import datetime

def func_1(title):
    now = datetime.datetime.now()
    print "hello, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond

def func_2(name):
    func_1('function func_2')
    now = datetime.datetime.now()
    print "Bye, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond

if __name__ == '__main__':
    p = Process(target=func_2, args=('bob',))
    p.start()
    p.join()

And I am getting a difference in microseconds. Is there any way to execute both at the exact same time? Any help would be appreciated.

Metronymic

The exact same time is theoretically impossible - the question is how big your error margin is allowed to be. In your case I guess you want to start them at the same microsecond? – Currie

Answer 1

On the computer the following was written on, this code consistently prints out the same timestamps:

#! /usr/bin/env python3
from multiprocessing import Barrier, Lock, Process
from time import time
from datetime import datetime

def main():
    synchronizer = Barrier(2)
    serializer = Lock()
    Process(target=test, args=(synchronizer, serializer)).start()
    Process(target=test, args=(synchronizer, serializer)).start()

def test(synchronizer, serializer):
    synchronizer.wait()
    now = time()
    with serializer:
        print(datetime.fromtimestamp(now))

if __name__ == '__main__':
    main()

Tip

A little late to the party but @NoctisSkytower why are you using the serializer here? I seem to get similar results just not using that at all. What is the motivation for it? – Yen

@Yen The print function does not construct a string before displaying it. There are several steps that are taken instead. The serializer is meant to ensure that all of those steps are atomic in nature. – Tip

Answer 2

This is (1) generally impossible (the "exact" part) and (2) not something that Python is good at. If you really need microsecond execution precision, use C or ASM, but an even closer way than COpython's answer would be busy-waiting in two different processes for an agreed start time:

from multiprocessing import Process
import os
import datetime
from time import time

def func_1(title):
    now = datetime.datetime.now()
    print "hello, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond

def func_2(name):
    now = datetime.datetime.now()
    print "Bye, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond

def start_f1(name):
    while time() < start_time: pass
    func_1(name)

def start_f2(name):
    while time() < start_time: pass
    func_2(name)        

if __name__ == '__main__':
    procs = []
    procs.append(Process(target=start_f2, args=('bob',)))
    procs.append(Process(target=start_f1, args=('sir',)))
    start_time = time() + 10
    map(lambda x: x.start(), procs)
    map(lambda x: x.join(), procs)

Currie

I've used this idea and I've gotten my processes to run consistently within 10ms of each other. Do you think that is the limit of this? any idea how to improve it even more? – Yen

@Yen writing a C extension could improve this a bit, as could using Barrier instead of the busy wait (see the accepted answer; python3 only). There's also things you could do on the OS side, e.g. tuning the process scheduler. But if a 10ms difference is not good enough for your use case I doubt this will help, as any real work you do afterwards won't stay synchronized (at least not without your intervention). What good is it to start on the same microsecond when the actual actions you want to synchronize can still be apart by milliseconds or more? – Currie

Answer 3

I am not sure if this will execute at exactly the same time, but I think that it will get you closer.

from multiprocessing import Process
import os
import datetime


def func_1(title):
    now = datetime.datetime.now()
    print "hello, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond


def func_2(name):
    now = datetime.datetime.now()
    print "Bye, world"
    print "Current second: %d" % now.second
    print "Current microsecond: %d" % now.microsecond


if __name__ == '__main__':
    procs = []
    procs.append(Process(target=func_2, args=('bob',)))
    procs.append(Process(target=func_1, args=('sir',)))
    map(lambda x: x.start(), procs)
    map(lambda x: x.join(), procs)

Irresolution

@COpython Sorry still there is 400ms diff. – Metronymic

@Mr.Calm, you could put both of your functions in a class, and get now in the constructor, then both functions would have the same timestamp... no other way to get it closer than that :-/ – Irresolution

Answer 4

CPythonis inherently single threaded (Google "Global Interpreter Lock"). To have even a theoretical chance you would need a multicore processor, but even then only an operating system operating at a very low level could do it and even then you would need special hardware.. What you are asking for is, in any practical sense, impossible.

Tangleberry

The question specifically mentions the multiprocessing module which sidesteps the GIL, so this part of your answer is invalid... – Currie

Maybe the GIL part is invalid, but the multiprocessing module relies on the OS scheduler, which will schedule in software and sequentually. You would need two processors that react to the same hardware interrupt which you aren't going to get without custom hardware. – Tangleberry

Answer 5

The multiprocessing with barrier solution didn't work for me, I blame my python environment. Got a FileNotFoundError from the multiprocessing semlock trying to rebuild the state.

Using the solution from l4mpi that used time() and a while loop to wait for an agreed upon time worked, but was off by a few milliseconds.

I found using the datetime.datetime.now() got me to within a few microseconds. This example same code from l4mpi (modified to work on my python, then swapped time() to datetime.datetime.now() based checking)

from multiprocessing import Process
import datetime

start_time = datetime.datetime.now().second + 10
start_time_micro = 500
def func_1(title):
    now = datetime.datetime.now()
    print(f"hello, world {title}")
    print( "Current second: %d" % now.second)
    print("Current microsecond: %d" % now.microsecond)

def func_2(name):
    now = datetime.datetime.now()
    print(f"Bye, world {name}")
    print("Current second: %d" % now.second)
    print("Current microsecond: %d" % now.microsecond)

def start_f1(name):
    while True:
        now = datetime.datetime.now()
        if now.second >= start_time and now.microsecond >= start_time_micro: 
            break
    func_1(name)

def start_f2(name):
    while True:
        now = datetime.datetime.now()
        if now.second >= start_time and now.microsecond >= start_time_micro: 
            break
    func_2(name)        

if __name__ == '__main__':
    procs = []
    print(f"setup: {start_time}")
    procs.append(Process(target=start_f2, args=('bob',), daemon=True))
    procs.append(Process(target=start_f1, args=('sir',), daemon=True))
    start_time = datetime.datetime.now().second + 10
    print(f"Starting {start_time}")
    for x in procs:
        x.start()

    for x in procs:
        x.join()
    print("Done")

Don't know why the

map(lambda x: x.start(), procs)
map(lambda x: x.join(), procs)

didn't work in my system, just converted to a for loop to avoid wasting time.

All in all it looked weird that one time function was used to print and check the difference between the two, while another time function was used to synchronize the two. So I wanted to test it out. Then the results were just interesting enough that I wanted to share with everyone else. Hope someone will find this helpful or informative.

Pedalfer