ZeroMQ hangs in a python multiprocessing class/object solution

Question

I'm trying to use ZeroMQ in Python (pyzmq) together with multiprocessing. As a minmal (not) working example I have a server- and a client-class which both inherit from multiprocessing.Process. The client as a child-process should send a message to the server-child-process which should print the message:

#mpzmq_class.py

from multiprocessing import Process
import zmq


class Server(Process):
    def __init__(self):
        super(Server, self).__init__()
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.PULL)
        self.socket.connect("tcp://localhost:6068")

    def run(self):
        msg = self.socket.recv_string()
        print(msg)


class Client(Process):
    def __init__(self):
        super(Client, self).__init__()
        self.ctx = zmq.Context()
        self.socket = self.ctx.socket(zmq.PUSH)
        self.socket.bind("tcp://*:6068")

    def run(self):
        msg = "Hello World!"
        self.socket.send_string(msg)

if __name__ == "__main__":
    s = Server()
    c = Client()
    s.start()
    c.start()
    s.join()
    c.join()

Now if I run this the server-process seems to hang at the receive-call msg = socket.receive_string(). In another (more complicated) case, it even hung at the socket.connect("...")-statement.

If I rewrite the script to use functions instead of classes/objects, it runs just fine:

# mpzmq_function.py

from multiprocessing import Process
import zmq


def server():
    ctx = zmq.Context()
    socket = ctx.socket(zmq.PULL)
    socket.connect("tcp://localhost:6068")
    msg = socket.recv_string()
    print(msg)


def client():
    ctx = zmq.Context()
    socket = ctx.socket(zmq.PUSH)
    socket.bind("tcp://*:6068")
    msg = "Hello World!"
    socket.send_string(msg)

if __name__ == "__main__":
    s = Process(target=server)
    c = Process(target=client)
    s.start()
    c.start()
    s.join()
    c.join()

Output:

paul@AP-X:~$ python3 mpzmq_function.py 
Hello World!

Can anybody help me with this? I guess it's something I didn't understand concerning the usage of multiprocessing.

Thank you!

Subminiaturize

Answer 1

I run into the same issue. I guess the problem is, that the run method has no access to the context object. Maybe it has something to do with the C implementation and the fact, that processes do not have shared memory. If instantiate the context in the run method, it works.

Here a working example:

#mpzmq_class.py

from multiprocessing import Process
import zmq


class Base(Process):
    """
    Inherit from Process and
    holds the zmq address.
    """
    def __init__(self, address):
        super().__init__()
        self.address = address


class Server(Base):
    def run(self):
        ctx = zmq.Context()
        socket = ctx.socket(zmq.PULL)
        socket.connect(self.address)
        msg = socket.recv_string()
        print(msg)


class Client(Base):
    def run(self):
        ctx = zmq.Context()
        socket = ctx.socket(zmq.PUSH)
        socket.bind(self.address)
        msg = "Hello World!"
        socket.send_string(msg)


if __name__ == "__main__":
    server_addr = "tcp://127.0.1:6068"
    client_addr = "tcp://*:6068"
    s = Server(server_addr)
    c = Client(client_addr)
    s.start()
    c.start()
    s.join()
    c.join()

I added a base class to demonstrate that you can still access normal Python objects from the run method. If you put the context object into the init Method, it won't work.

Beget

You're right, that's exactly how I managed to solve this problem, just couldn't manage to answer to myself until now. For reasons unknown all ØMQ has to be instantiated during runtime of the process. Furthermore I ended up calling init_zmq()s where I redefine attributes of the objects linked to ØMQ (i.e. context, sockets) earlier to be defined as None. This is a little circuitous but I think it's alright. – Subminiaturize

I fell into the same trap, but it does state in the chapter zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ how to write multithreaded and multiprocess applications: "Create one ZeroMQ context at the start of your process, and pass that to all threads that you want to connect via inproc sockets." Note the word 'process', it means that the zmq context is created in the process's run() method, thereafter the context (but not sockets!) can be shared by all threads in that process. If threading is also used create sockets in the threads' run() method. – Wentzel