Make gRPC messages JSON serializable

Question

I have a gRPC definition that contains repeated scalar values.

For example

message Request {
  bool dryrun = 1;   
  bool verbose = 2;
  bool enabled = 3;
  float ratio = 4;
  int32 count = 5;
  string somevar = 6;
  bool withcondition = 7;
  bool nocondition = 8;
  float portion = 9;
  int32 population = 10;
  string xvars = 11;
  string singleoption = 12;
  repeated int32 multioptions = 13;
}

I noticed that when this request gets sent to a Python server, mylist is converted to a RepeatedScalarContainer, which is not JSON serializable. Some SO questions suggest that I use

from google.protobuf.json_format import MessageToJson
serialized = MessageToJson(original)

to convert a protobuf message to a valid JSON object. But in my case, doing the above on the received Python object gives me a truncated one

{
  "enabled": true,
  "ratio": 0.5,
  "count": 100,
  "somevar": "/input",
  "withcondition": true,
  "singleoption": "option1",
  "multioptions": [
    1,
    3
  ]
}

You can see that a bunch of message attributes are missing.

As a workaround, I could still extract the attributes of the received object to recover the data and convert them to JSON serializable by brute-force, but this seems ugly and flaky. Here is my way

    def handle_request(request):
        all_attrs = dir(request)
        i_last_private_attr = next((a for a, attr in enumerate(reversed(all_attrs)) if attr.startswith('_')), None)
        if not i_last_private_attr:
            _logger.error('Failed to find any request')
            return
        i_first_arg_attr = len(all_attrs) - i_last_private_attr
        req_dict = {attr: getattr(request, attr) for attr in all_attrs[i_first_arg_attr:]}
        for k, v in req_dict.items():
            if isinstance(v, Iterable) and not isinstance(v, (dict, list, tuple, str)):
                req_dict[k] = list(req_dict[k])

How should I make the received raw gRPC request work like a Python object containing only builtin primitive value types?

Unpleasantness

for efficiency grpc normally doesn't transmit fields which contain their default values. That's why they disappear. The receiver will restore those fields in the receive message. – Meghannmegiddo

Answer 1

Solved it myself.

There is an option including_default_value_fields with json_format.MessageToJson(). When set to False, all default fields are not printed to JSON, although they are physically sent through the wire.

Unfortunately, the option was turned off by default.

After setting it to True, the problem is gone.

Unpleasantness