How can i write my own RPC Implementation for Protocol Buffers utilizing ZeroMQ

Question

According to the Google Protocol Buffers documentation under 'Defining Services' they say,

it's also possible to use protocol buffers with your own RPC implementation.

To my understanding, Protocol Buffers does not implement RPC natively. Instead, they provide a series of abstract interfaces that must be implemented by the user (Thats me!). So I want to implement these abstract interfaces utilizing ZeroMQ for network communication.

I'm trying to create an RPC implementation using ZeroMQ because the project i'm working on already implements ZeroMQ for basic messaging (Hence why I'm not using gRPC, as the documentation recommends).

After reading through the proto documentation thoroughly, i found that I have to implement the abstract interfaces RpcChannel and RpcController for my own implementation.

I've constructed a minimalized example of where I'm currently at with my RPC Implementation

.proto file: Omitted SearchRequest and SearchResponse schema for brevity

service SearchService {
    rpc Search (SearchRequest) returns (SearchResponse);
}

SearchServiceImpl.h:

class SearchServiceImpl : public SearchService {
 public:
  void Search(google::protobuf::RpcController *controller,
                    const SearchRequest *request,
                    SearchResponse *response,
                    google::protobuf::Closure *done) override {
    // Static function that processes the request and gets the result
    SearchResponse res = GetSearchResult(request);

    // Call the callback function
    if (done != NULL) {
    done->Run();
    }
    }
  }
};

MyRPCController.h:

class MyRPCController : public google::protobuf::RpcController {
 public:
    MyRPCController();

    void Reset() override;

    bool Failed() const override;

    std::string ErrorText() const override;

    void StartCancel() override;

    void SetFailed(const std::string &reason) override;

    bool IsCanceled() const override;

    void NotifyOnCancel(google::protobuf::Closure *callback) override;
 private:
  bool failed_;
  std::string message_;
};

MyRPCController.cpp - Based off of this

void MyRPCController::Reset() {  failed_ = false; }

bool MyRPCController::Failed() const { return failed_; }

std::string MyRPCController::ErrorText() const { return message_; }

void MyRPCController::StartCancel() { }

void MyRPCController::SetFailed(const std::string &reason) {
  failed_ = true;
  message_ = reason;
}

bool MyRPCController::IsCanceled() const { return false; }

void MyRPCController::NotifyOnCancel(google::protobuf::Closure *callback) { }

MyRPCController::ChiRpcController() : RpcController() { Reset(); }

MyRpcChannel.h:

class MyRPCChannel: public google::protobuf::RpcChannel {
 public:
    void CallMethod(const google::protobuf::MethodDescriptor *method, google::protobuf::RpcController *controller,
                    const google::protobuf::Message *request, google::protobuf::Message *response,
                    google::protobuf::Closure *done) override;
};

Questions I have with my example thus far:

• Where do I fit ZeroMQ into this?
  • It seems like it should be going into RPCChannel, because in the examples i see (See 3rd code block here), they pass a string that has the ports to bind to (i.e. MyRpcChannel channel("rpc:hostname:1234/myservice");)
• I'm concerned with my RPCController implementation, it seems too simple. Should more be going here?
• How do i implement RPCChannel, it seems very similar to the SearchServiceImpl. The 1 virtual function in these classes has a very similar method signature, except it's generic.

Here's some other Stack Overflow questions I came across that had some helpful information on the topic:

1. Protobuf-Net: implementing server, rpc controller and rpc channel - This is where i found the example for the RPCController implementation.
2. Using Protocol Buffers for implementing RPC in ZeroMQ - This answer is interesting because in the top answer, is seems that they're recommending against using Protobufs built in RPC formatting for the .proto file.
   • I also noticed this same notion in this file, in a repository called libpbrpc which seemed like a good source for example code
3. Can I/Should I be using an existing implementation such as RPCZ?

Thank you for your help. I hope I gave enough information and was clear in what I'm looking for. Please let me know if something is unclear or lacking in information. I'd be happy to edit the question accordingly.

Dekow

Same problem here, what did you end up doing? – Domiciliary

Answer 1

• ZeroMQ provides a low-level API for network communication based on messages that can contain any data.
• ProtoBuffers is a library that encodes structured data as compressed binary data and decodes such data.
• gRPC is a RPC framework that generates code for network communication based RPC services with functions that exchange data as ProtoBuffers data.

Both ZeroMQ and gRPC provides support for network communication but in different ways. You have to chose either ZeroMQ, either gRPC for network communication. If you choose ZeroMQ, messages can be encoded using ProtoBuffers exchanging binary structured data.

The main point is ProtoBuffers library allows variant records (similar to C/C++ unions) to be encoded and decoded that can fully emulate the functionality provided by RPC services having functions exchanging ProtoBuffers messages.

So the options are:

1. Use ZeroMQ with send and receive primitives and ProtoBuffers encoded variant messages that can contain various sub-messages, like

union Request
{
  byte msgType;
  MessageType1 msg1;
  MessageType2 msg2;
  MessageType3 msg3;
}

union Response
{
  byte msgType;
  MessageType3 msg1;
  MessageType4 msg2;
  MessageType5 msg3;
}

send(Request request);
receive(Response response);

2. Use gRPC generating a service with functions, like

service MyService 
{
  rpc function1(MessageType1) returns (Response);
  rpc function2(MessageType2) returns (Response);
  rpc function3(MessageType3) returns (Response);

  rpc functionN(MessageType3) returns (MessageType5);
}

(here it's possible to use many many combinations)

3. Use just a single-function gRPC service, like

service MyService 
{
    rpc function(Request) returns (Response);
}

The option could depend on

• preferred target for client: ZeroMQ or gRPC based client
• performance reasons comparing ZeroMQ vs gRPC based service
• specific features like how subscription is used/handled in ZeroMQ vs gRPC based service and client (see How to design publish-subscribe pattern properly in grpc?)

For the 1st option, you have to do a lot of stuff comparing to 2nd option. You have to match the type of message sent with the types of expected messages to be received.

The 2nd option would allow an easier/faster understanding of functionality of the service provided if the somebody else will develop the client.

For developing a RPC service on top on ZeroMQ I would define such .proto file specifying the functions, parameters (all possible input and output parameters) and errors like this:

enum Function 
{
    F1 = 0;
    F2 = 1;
    F3 = 2;
}

enum Error 
{
    E1 = 0;
    E2 = 1;
    E3 = 2;
}

message Request
{ 
    required Function function = 1;
    repeated Input data = 2;
}

message Response
{ 
    required Function function = 1;
    required Error error = 2;
    repeated Output data = 3;
}

message Input
{ 
    optional Input1 data1 = 1;
    optional Input2 data2 = 2;
    ...
    optional InputN dataN = n;
}

message Output
{ 
    optional Output1 data1 = 1;
    optional Output2 data2 = 2;
    ...
    optional OutputN dataN = n;
}

message Message
{
   repeated Request requests;
   repeated Response responses;
}

and depending on function id, at run-time the number and the types of parameters have to be checked.

Hourihan

Hi Flaviu, thanks for taking the time to write all this out. There's a lot of good information here. I'm kind of stuck with using ZeroMQ along with Protocol Buffers rather than gRPC because our project already uses ZeroMQ and they don't want to implement another network. So i've been trying to explore how to properly integrate the abstract interfaces for RPC that Protocol Buffers provides, and how to implement them with ZeroMQ. – Dekow

Hi and welcome. So you want to have a RPC service on top of ZeroMQ. For that 2 theorethical protocols have to be defined, one to specify RPC stuff (function id, no of params, errors that can occur) and one to specify the messages (parameters) exchanged by RPC functions. Please have a look at XML-RPC (en.wikipedia.org/wiki/XML-RPC) and JSON-RPC (en.wikipedia.org/wiki/JSON-RPC). Based on such RPC definitions you can write a .proto file covering RPC and messages stuff - I guess this is the difficul part. Then generate ProtoBuffers code and then integrate it in the ZeroMQ project. – Hourihan

Thanks for your reply. To my understanding, instead of using Xml-RPC or Json RPC, i thought i could achieve the same sort of function leveraging Protobufs. Reading through their "Services" documentation it sounds like that's the case, which is what i'm trying to accomplish. – Dekow

"or that 2 theorethical protocols have to be defined, one to specify RPC stuff (function id, no of params, errors that can occur) and one to specify the messages (parameters) exchanged by RPC functions." From this, it sounds like this problem could be solved using Protobuf schemas, i could be wrong though. Thanks again for taking the time to respond, i really appreciate the information. – Dekow

I added a possible RPC as Protobuffers schema. Welcome ;) – Hourihan

Thanks for the extra information. I've done some thinking and based on the example you provided, I think i came up with a valid solution. Originally i was thinking about leveraging the RPC syntax that protobuf provides and somehow utilizing that, but i was getting lost in the details. My new idea, based on your example. Protobuf messages can simply contain a function name, in my code, ill have a map that maps a string to a certain function. When a message is received, it will execute the appropriate function. It seems a lot simpler than what I was originally imagining. – Dekow

Glad to hear that. Regarding performance, it would be faster to use ids (enums) for functions instead of strings. When texts associated to enums are needed, I used github.com/cflaviu/decl_enum. Also FlatBuffers (google.github.io/flatbuffers) could be used instead of ProtoBuffers. They are very similar to ProtoBufffers. FlatBuffers do not compress data, thus data is sent/received faster. – Hourihan