How to I transfer an image(opencv Matrix/numpy array) from c++ publisher to python sender via ZeroMQ?

Question

I know how to send a string message from c++ to python via zeromq.

Here's the code for sending a string message I know :

C++ sender code :

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://localhost:5563");
std::string message = "Hello from sender";
const char *message_char = message.c_str();
zmq_send(publisher, message_char, strlen(message_char), ZMQ_NOBLOCK);

Python receiver code :

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://*:5563")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
message = receiver.recv_string()

What I want is to send an image from c++ zeromq publisher to python receiver.

Navicert

Maybe use a std::vector<uint8_t> rather than std::string? – Fregoso

How do you have the image stored in c++? – Toddle

@Toddle I've used the library opencv for handling images. Its reads image in form of Matrix object. – Navicert

@Rohit You should write those things at the beginning of your question since c++ does not offer by default the handling of images so you will always need to use an external library. :-) – Toddle

@Toddle Thanks for pointing out. I've declared use of opencv library in the question itself. – Navicert

Answer 1

Disclaimer : Answering my own question, so that others shall not stuck where I did.

So lets get started.

What is Zero MQ ?

ZeroMQ is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.

Before, we get started here are the basics :

Protocol/Library used : ZeroMQ

Publisher : C++ oriented

Subsciber : Python oriented

---

Sending String/char array message via ZeroMQ :

C++ Publisher :-

// Setting up ZMQ context & socket variables
void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB); 
int bind = zmq_bind(publisher, "tcp://*:9000");
std::string message = "Hello from sender";
const char *message_char = message.c_str(); // Converting c++ string to char array
// Sending char array via ZMQ
zmq_send(publisher, message_char, strlen(message_char), ZMQ_NOBLOCK);

Python Subscriber :-

// Setting up ZMQ context & socket variables
context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
// Subscribing to start receiving messages
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
message = receiver.recv_string()

---

Sending Image/ndarray array message via ZeroMQ :

For handling images, opencv is an awesome library. Simple, easy to code & lightning fast.

C++ Publisher :-

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://*:9000");

// Reading the image through opencv package
cv::Mat image = cv::imread("C:/Users/rohit/Desktop/sample.bmp", CV_LOAD_IMAGE_GRAYSCALE );
int height = image.rows;
int width = image.cols;
zmq_send(publisher, image.data, (height*width*sizeof(UINT8)), ZMQ_NOBLOCK);

In above code, image is read as a grayscale image, you can read 3-channel (RGB) image as well, by passing appropriate parameters in opencv's imread method.

Also remember to modify the size (3rd parameter in zmq_send function call) accordingly.

Python Subscriber :-

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
// Receiving image in bytes
image_bytes = receiver.recv()
int width = 4096; // My image width
int height = 4096; // My image height
// Converting bytes data to ndarray
image = numpy.frombuffer(image_byte, dtype=uint8).reshape((width, height))

TO DO / IMPROVEMENT : You can also pass the image size from the c++ publisher along with the image data. So, that image can be reshaped accordingly at python side.

ZMQ_SNDMORE flag comes handy here

Just add another zmq_send statement at c++ side.

zmq_send(publisher, img_height, strlen(img_height), ZMQ_SNDMORE)
zmq_send(publisher, img_width, strlen(img_width), ZMQ_SNDMORE)
zmq_send(publisher, image.data, (height*width*sizeof(UINT8)), ZMQ_NOBLOCK);

Similarly, add corresponding receiving statements at python end.

height = receiver.recv_string(ZMQ_RCVMORE)
width = receiver.recv_string(ZMQ_RCVMORE)
image_bytes = receiver.recv()

---

Another Improvement

Thanks @Mark Setchell for pointing out an improvement.

Sending opencv Matrix of large size directly over network can be costly. A better approach would be to encode the image before sending over the network.

C++ Publisher :-

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://*:9000");

// Reading the image through opencv package
cv::Mat image = cv::imread("C:/Users/rohit/Desktop/sample.bmp", CV_LOAD_IMAGE_GRAYSCALE );
int height = image.rows;
int width = image.cols;
cv::vector<uchar> buffer;
cv::imencode(".jpg", image, buffer);
zmq_send(publisher, buffer.data(), buffer.size(), ZMQ_NOBLOCK);

Python Subscriber :-

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
// Receiving image in bytes
image_bytes = receiver.recv()
// Decoding the image -- Python's PIL.Image library is used for decoding
image = numpy.array(Image.open(io.BytesIO(image_byte)))

Navicert

Sending a 16MB image uncompressed is potentially pretty wasteful of network bandwidth! Maybe consider using cv2.imencode() to convert to JPEG or PNG in memory before sending and it will likely be 200kB instead of 16MB. – Samford

@MarkSetchell Thanks Mark for point out, I've updated the code with image encoding/decoding integrated at both c++ & python ends. – Navicert

May you provide the inverse that is sending an image (encoded) from python client to c++ server? – Geter

An example of encoding before sending can be seen in this answer – Kovrov