file transfer using gRPC

Question

What is the standard way to use gRPC and protocol buffer to transfer file or images from one process to another? The file is from 1MB to 6MB.

Basically I want to know how to implement sender / receiver code pairs for python, C++ and golang.

http://github.com/johanbrandhorst/chunker

This is the closest I can find so far for golang. I am thinking whether it is extendable to python and c++. A more general approach with less dependencies is preferred.

I am creating multiservices to exchange images btw machines/processes via IPC message passing. Another option is to use shared memory. I am not sure whether Redis can store images

Banshee

Answer 1

I haven't done that before, but I think this boils down to the types the protocol buffer language supports.

My understanding is that you can use the bytes type since it may contain any arbitrary sequence of bytes. Encode the content before storing it, assign it and "ship it over"!

Have a look here for the scalar types.

Another approach would be to store the file in a location the "services" involved can read — and maybe write if required for some of them. Then send a gRPC message to the "interested" services with the information required to access (or do anything else with) the file(s) in question.

Meras

Answer 2

For small-scale files (say, less than 1 MiB) putting the entire file in a single message (as bytes) is very straight-forward. For larger files, chunking the files into parts and using a stream can be natural; see my answer for a similar question.

Panzer

Answer 3

To the best of my knowledge, sending file of size larger than 1MB using protocol buffer might not be the best way. Sending the file chunk by chunk via byte string via socket is a better approach.

How to send files in "chunks" by socket?

Banshee