How is a multiprocessing.Event passed to a child process under the hood?
Asked Answered
S

1

0

According to this SO answer, when a multiprocessing.Queue is passed to a child process, what is actually sent is a file descriptor (or handle) obtained from pipe, instead of pickling the Queue object itself.

How does this work with multiprocessing.Event, e.g. when doing a

cancel_event = multiprocessing.Event()
process = multiprocessing.Process(target=worker_function, args=(cancel_event, ))

I would assume that Event must also have some OS-related thing which is sent to the child process. Is this also a handle for a pipe like with queues?

Sacculus answered 28/10, 2023 at 16:11 Comment(0)
E
1

The Event class is a very simple extension of a Condition along with an additional Semaphore. A Condition contains a Lock and a couple more semaphores, and a Lock is just a Semaphore with a maximum value of only 1, so the question can really be boiled down to how is a Semaphore passed to another process...


When creating a process using a Spawning context or when sending a lock over a queue or pipe, it is serialized using pickle:

The base class for both Semaphore and Lock is SemLock, which defines the __getstate__ and __setstate__ methods in order to make it able to be sent across a pipe using the pickle protocol.

On the sending side, __getstate__ will obtain a handle to the lock which can be sent. On Windows one cannot simply take the handle returned by CreateSemaphore and use it in another process, so it is instead first duplicated with DuplicateHandle giving it the target process where it will be used. On *nix the handle is simply copied.

On the receiving side, the handle number is used along with a unique name and some other attributes which are copied over in order to rebuild the semaphore. This is a pretty simple process of unpacking values into a new SemLockObject struct, which contains the handle to the underlying OS level semaphore.


When creating a process using a Forking context

The new process gets a complete copy of the memory of the parent process, including the objects containing references to the semaphore handles (see: forking and process copy-on-write). As windows is not involved here, the simple copying of the handle value is valid.

Engineer answered 31/10, 2023 at 5:14 Comment(2)
Thanks a lot for the detailed answer. So basically it is boiled down to passing an integer value that is known/managed by the OS. Did I understand that correctly?Sacculus
@Sacculus yes, although there is some "permission to use that integer" going on too... In this case windows is more restrictive, requiring extra steps to make a handle sharable.Engineer

© 2022 - 2024 — McMap. All rights reserved.