MultiProcessing Pipe recv blocks even when child process is defunct

Question

Reading several questions on this topic I understand now that the child process inherits the file descriptors from the parent process. Which will make it more difficult for a child to receive an EOFError when a parent closes the connection.

But my situation is the other way around, and I do not understand the problem I am facing.

I have a parent process that starts a child process, and gives it access to one end of the Pipe connection I created. Now when the child process is done, malfunctions or whatever, everything is stopped and the connection is closed. At this point the child process shows to be defunct.

Then I expect the parent process' connection to throw an EOFError on the blocking recv call. But instead it just sits there and waits.

What am I missing here?

EDIT

I think this example represents the problem:

from multiprocessing import Process, Pipe
from threading import Thread
import time

class Parent(object):

    def __init__(self):
        self.parent_conn, child_conn = Pipe()
        self.child = Process(target=Child, args=(child_conn,))
        self.child.start()        

        def recv():
            try:
                self.parent_conn.recv()
            except EOFError:
                print "EOF"
            except:
                print "something else"

        # Does not work
        recv()

        # Works fine
        t = Thread(target=recv)
        t.setDaemon(True)
        t.start()

    def close(self):
        self.parent_conn.close()
        self.child.join()

class Child(object):

    def __init__(self, conn):
        conn.close()

if __name__ == "__main__":
    p = Parent()
    time.sleep(1)
    p.close()

If I do use the separate thread, the parent is allowed to close its own connection and everything works fine. (Note that you still need to know somehow that the child is done for, to do this) Instead if I call recv directly it will block obviously, but I would suspect it to raise an EOFError as soon as the child process closes its connection. But it does not. Can anyone clarify?

Parlous

Could you provide a minimal code example that demonstrates the issue? – Hanoi

@J.F.Sebastian, I provided an example. To show that recv keeps blocking. I also provided the second option which shows that it can easily close down if made possible. – Parlous

add child_conn.close() after self.child.start(). It is idiomatic for working with pipes to close unused ends. Also (optionally) provide duplex=False parameter. If .close() works for you; please, post it as your own answer for others benefit. – Hanoi

The things is, I don't know beforehand whether it is going to close right away.. Normally the child should be able to send and receive. Furthermore I still do not get why this won't work, as is. – Parlous

I've updated the answer, to address your latest edit. – Hanoi

There is nothing in the pipe, so the rev() is blocking, waiting for the sender to send something. You can add conn.send('hello') in the __init__ function of Child class to test it. – Sympathin

Answer 1

Add child_conn.close() after self.child.start(). It is idiomatic for working with pipes to close unused ends. Also (optionally) provide duplex=False parameter.

The things is, I don't know beforehand whether it is going to close right away.. Normally the child should be able to send and receive. Furthermore I still do not get why this won't work, as is.

1. child_conn.close() in the parent doesn't mean that child should close its end immediately
2. parent_conn.recv doesn't return as long as there is a chance that somebody will child_conn.send(). If child_conn is opened (in child or parent) then there is a chance

If I do use the separate thread, the parent is allowed to close its own connection and everything works fine. Note that you still need to know somehow that the child is done for, to do this

You don't need to know it. You can close as soon as the connection is opened in the child. Calling child_conn.close() in the parent after self.child.start() is fine whatever child does.

Could you explain the duplex option a little bit more also?

duplex=False means that the pipe is unidirectional i.e., you can only call parent_conn.recv() and child_conn.send(). Otherwise it is bidirectional and both connections support send/recv.

Hanoi

Yes, it works :). Took me a few minutes to realise that I of course shouldn't close the connection until after I had passed it to the child..:p Could you explain the duplex option a little bit more also? I think it means that only one end is allowed to send, if set false? – Parlous

@Oxidator: I've added description of duplex parameter. – Hanoi