Protobuf 3.0 Any Type pack/unpack

Question

I would like to know how to transform a Protobuf Any Type to the original Protobuf message type and vice versa. In Java from Message to Any is easy:

Any.Builder anyBuilder = Any.newBuilder().mergeFrom(protoMess.build());

But how can I parse that Any back to the originial message (e.g. to the type of "protoMess")? I could probably parse everything on a stream just to read it back in, but that's not what I want. I want to have some transformation like this:

ProtoMess.MessData.Builder protoMessBuilder = (ProtoMess.MessData.Builder) transformToMessageBuilder(anyBuilder)

How can I achieve that? Is it already implemented for Java? The Protobuf Language Guide says there were pack and unpack methods, but there are none in Java. Thank you in Advance :)

Renie

Answer 1

The answer might be a bit late but maybe this still helps someone.

In the current version of Protocol Buffers 3 pack and unpack are available in Java.

In your example packing can be done like:

Any anyMessage = Any.pack(protoMess.build()));

And unpacking like:

ProtoMess protoMess = anyMessage.unpack(ProtoMess.class);

---

Here is also a full example for handling Protocol Buffers messages with nested Any messages:

ProtocolBuffers Files

A simple Protocol Buffers file with a nested Any message could look like:

syntax = "proto3";

import "google/protobuf/any.proto";

message ParentMessage {
  string text = 1;
  google.protobuf.Any childMessage = 2;
}

A possible nested message could then be:

syntax = "proto3";

message ChildMessage {
  string text = 1;
}

Packing

To build the full message the following function can be used:

public ParentMessage createMessage() {
    // Create child message
    ChildMessage.Builder childMessageBuilder = ChildMessage.newBuilder();
    childMessageBuilder.setText("Child Text");
    // Create parent message
    ParentMessage.Builder parentMessageBuilder = ParentMessage.newBuilder();
    parentMessageBuilder.setText("Parent Text");
    parentMessageBuilder.setChildMessage(Any.pack(childMessageBuilder.build()));
    // Return message
    return parentMessageBuilder.build();
}

Unpacking

To read the child message from the parent message the following function can be used:

public ChildMessage readChildMessage(ParentMessage parentMessage) {
    try {
        return parentMessage.getChildMessage().unpack(ChildMessage.class);
    } catch (InvalidProtocolBufferException e) {
        e.printStackTrace();
        return null;
    }
}

EDIT:

If your packed messages can have different Types, you can read out the typeUrl and use reflection to unpack the message. Assuming you have the child messages ChildMessage1 and ChildMessage2 you can do the following:

@SuppressWarnings("unchecked")
public Message readChildMessage(ParentMessage parentMessage) {
    try {
        Any childMessage = parentMessage.getChildMessage();
        String clazzName = childMessage.getTypeUrl().split("/")[1];
        String clazzPackage = String.format("package.%s", clazzName);
        Class<Message> clazz = (Class<Message>) Class.forName(clazzPackage);
        return childMessage.unpack(clazz);
    } catch (ClassNotFoundException | InvalidProtocolBufferException e) {
        e.printStackTrace();
        return null;
    }
}

For further processing, you could determine the type of the message with instanceof, which is not very efficient. If you want to get a message of a certain type, you should compare the typeUrl directly:

public ChildMessage1 readChildMessage(ParentMessage parentMessage) {
    try {
        Any childMessage = parentMessage.getChildMessage();
        String clazzName = childMessage.getTypeUrl().split("/")[1];
        if (clazzName.equals("ChildMessage1")) {
            return childMessage.unpack("ChildMessage1.class");
        }
        return null
    } catch (InvalidProtocolBufferException e) {
        e.printStackTrace();
        return null;
    }
}

Estate

Is there no other way than this readChildMessage? What if I there are dozens of different messages that could come in? Just add in new try-catch blocks? Even switch-case and the like is absolutely unacceptable. – Lanfranc

Good question, I forgot to add that you can get the name of the packed message as typeURL. This allows to unpack any message via reflection or directly decide what to do with the message. I added two examples to my answer, I hope this helps. – Estate

Great! It helped me! – Witless

childMessage.getTypeUrl().split("/")[1]; won't give the java package name right? It will give the proto package and hence will not work. – Amulet

@AdityaJoshee that's correct. Ideally, I'd like to get the canonical name of the class, but you get the package of the proto files. That's quite annoying. – Sideslip

My problem is that I have hundred of possible types and I cannot really write a (X == Y)-like condition for each of them. – Sideslip

Answer 2

Just to add information in case someone has the same problem ... Currently to unpack you have to do (c# .netcore 3.1 Google.Protobuf 3.11.4)

Foo myobject = anyMessage.Unpack<Foo>();

Caiaphas

Answer 3

I know this question is very old, but it still came up when I was looking for the answer. Using @sundance answer I had to answer do this a little differently. The problem being that the actual message was a subclass of the actual class. So it required a $.

    for(Any x : in.getDetailsList()){
            try{
                String clazzName = x.getTypeUrl().split("/")[1];
                String[] split_name = clazzName.split("\\.");
                String nameClass = String.join(".", Arrays.copyOfRange(split_name, 0, split_name.length - 1)) + "$" + split_name[split_name.length-1];
                Class<Message> clazz = (Class<Message>) Class.forName(nameClass);

                System.out.println(x.unpack(clazz));

            } catch (Exception e){
                e.printStackTrace();
            }
        } 

With this being the definition of my proto messages

    syntax = "proto3";
    package cb_grpc.msg.Main;

    service QueryService {
        rpc anyService (AnyID) returns (QueryResponse) {}
    }

    enum Buckets {
        main = 0;
        txn = 1;
        hxn = 2;
       }

    message QueryResponse{
        string content = 1;
        string code = 2;
    }

    message AnyID {
        Buckets bucket = 1;
        string docID = 2;
        repeated google.protobuf.Any details = 3;
    }

and

    syntax = "proto3";
    package org.querc.cb_grpc.msg.database;

    option java_package = "org.querc.cb_grpc.msg";
    option java_outer_classname = "database";

    message TxnLog {
        string doc_id = 1;
        repeated string changes = 2;
    } 

Gilson

Answer 4

This is what I did in Android Java on my project as a pack function:

String name = message.getClass().getSimpleName();
String packageName = message.getClass().getPackage().getName();   //needs to be the same as your proto package
String fullName = packageName + '.'+ name;
String typeURL = GetTypeUrl(fullName);
    
Any any = Any.newBuilder()
            .setValue(message.toByteString())
            .setTypeUrl(typeURL)
            .build();

private String GetTypeUrl(String fullName) {
    String prefix = "type.googleapis.com";
   return prefix + "/" + fullName;
}

Nolanolan