How does ZeroMQ REQ/REP handle multiple clients?

Question

I started to use ZeroMQ for IPC and made a simple echo-client/server and I'm surprised about one thing. Here is the C++ code (using zmq.hpp and zmq_addon.hpp).

Server:

zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("ipc:///tmp/machine-1");
while (1) {
    zmq::multipart_t m;
    m.recv(socket);
    int i = m.poptyp<int>();
    i++;
    m.addtyp<int>(i);
    m.send(socket);
}

Client:

zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REQ);

socket.connect("ipc:///tmp/machine-1");

int i = 0;
while (1) {
    int save = i;
    zmq::multipart_t m;
    m.addtyp<int>(i);
    m.send(socket);
    m.recv(socket);

    i = m.poptyp<int>();

    if (i != (save + 1))
        break;

    if ((i % 100000) == 0)
        std::cerr << "i : " << i<< "\n";
}

I works as expected. The client is sending an int, the server does plus one and sends it back.

Now the magic I don't understand: I realized, that I can run the client several times in parallel and it continues to works, for each client correctly.

The check comparing save+1 to i is always OK.

How does ZMQ handles the concurrency problem on the server side? How does it know to which client the response has to be send back?

There is this question on SO, but it doesn't answer my question: ZeroMQ REQ/REP on ipc:// and concurrency

Fusty

Relevant excerpt: "Doing any other sequence (e.g., sending two messages in a row) will result in a return code of -1 from the send or recv call. Similarly, the [server] issues zmq_recv() and then zmq_send() in that order". So i suspect this means we can not receive a later message before the previous message was replied to (even if work in between receive and send takes long). At least not with this basic mechanism – Orchestral

Answer 1

Per the zeromq docs, when you call REP.recv() in the server it will return a message from an enqueued REQ (client) socket. If there are multiple clients connected it will use a fair-queue policy to choose one. When you call REP.send() to reply, the REP socket always sends the response to the corresponding REQ client.

That is the "magic" - the REP socket takes care of sending the response to the correct client. If the client has disconnected it just drops the reply message.

The docs may be clearer than my explanation:

ZMQ_REP: A socket of type ZMQ_REP is used by a service to receive requests from and send replies to a client. This socket type allows only an alternating sequence of zmq_recv(request) and subsequent zmq_send(reply) calls. Each request received is fair-queued from among all clients, and each reply sent is routed to the client that issued the last request. If the original requester does not exist any more the reply is silently discarded.

Change

I only read the guide, you're right in the documentation of zmq_connect it is nicely described. The key is, that a ZMQ_REQ/ZMQ_RES-connection also requires a certain recv/send-pattern. So, they can lock access to a socket when recv is called, and release it when send is called (vice-versa for the other side) – Fusty

Answer 2

The glib (and not very useful) answer is, it works because they've written it that way.

Longer answer: what the ZMQ team have done is implement their own message passing protocol (zmtp) on top of stream connections (ipc pipes, sockets, etc). As well as passing and demarcating messages, they have put features into this protocol specifically to support different patterns like REQ/REP, PUB/SUB, fair queuing, etc. To make it work there is a zmq library thread(s) running that handles all the zmtp activity in the background, and you interact with this thread via calls to zmq_send, zmq_poll, etc. The use of zmtp means that the program at the other end of a socket must also be speaking zmtp; nothing useful happens if one end is using libzmq and the other is simply open the raw socket for itself.

It is a very useful piece of code indeed.

In my opinion this is definitely the way to go. ZMQ successfully abstracts the idea of a connection between two thread of execution to the point one no long cares whether they're on the same machine, in the same process, separated by a network connection, etc. That makes for easy application development - anything can go anywhere (ignoring issues related to network speeds and latencies).

I gather that you can even bind a zmq socket to two different transports, for example both ipc and tcp. That's super useful!

Cancellation