ZeroMQ: REQ/REP with large messages

Question

I am trying to send large messages ( 300 MB ) over a REP-socket.
There are multiple clients, each connecting to the server via a REQ-socket. The server creates a dedicated socket for every client, waits for the request ( containing an identifier ) and sends the message in two parts using ZMQ_SENDMORE:

1. metadata (~ 1 KB)
2. data (~ 300 MB)

After that the REP-socket is immediately closed. The Context()-instance is then closed in a parent thread. ZMQ_LINGER time of the socket is left at default ( infinite ). Sometimes the metadata is sent, but not the image data.
I traced the calls within ZeroMQ and found that the internal (windows) socket is closed before the message data has been sent to the network. I thought that zmq_term() would block as long as there are unsent messages in the queue.
As a workaround I changed the clients to send another request message after receiving the data as an acknowledgement. This works quite well, but I'm not quite sure if I have addressed the fundamental problem.

The ZeroMQ version is 4.0.4. I am using the C++ bindings. Server and clients all run on Windows ( 7 and 10 ).

Edmonds

What transport-class do you use for the REQ/REP sessions? You might want to know, that StackOverflow encourages users to post a so called MCVE -- a M -inimum C -omplete V -erifiable E -xample of code ( + data, where necessary ) that reproduces the problem under review. Anyway, enjoy this great Community of Knowledge. – Kreisler

Do you have a small sample which reproduces the problem? If so, I would encourage you to also post it on the zeroMQ mailing list. – Dubbin

Answer 1

I know that this is NOT the answer you want, but I experienced something similar with ZeroMQ on Linux. The underlying Linux socket closed and some messages never made it through.

In my use case, I hold the server open until the user gives me some keyboard input. It's a hack, but for my use case, it gives me what I want.

You can also sleep for a second or two before you close the socket. Once again, not pretty, but it might be a quick fix until this gets patched.

Edited to add: I am also sending multi-part messages and one of my message parts is fairly large (~200MB).

Vivianaviviane

Pete, you might already know, that ZeroMQ gives you much more controls for situations alike this. One can create several/many-sockets between relevant peers and explicitly signal ( yes, send soft-signals SIG_TERM, SIG_KILL et al ) between processes and thus create an explicit request for remote .close() of the communication channels' transport infrastructure if and once needed ( let processes be themselves smart and explicit, rather than relying on "bloody"-hacks once one can keep tidy & Finite-State-Automaton alike explicit control of valuable resources the system relies on ). – Kreisler

@Vivianaviviane Do you have an example to reproduce the issue? It will greatly improve the chances that it gets fixed quickly if you can provide a code sample and post it on the mailing list or create an issue on Github. – Dubbin