How to handle the generic type Object with protocol buffers, in the .proto file?

Question

I've spent some time looking for some alternative to handle generic objects, I've seen questions similar to mine, but not as specific I suppose? Protocol buffers has multiple scalar types that I can use, however they are mostly primitive. I want my message to be flexible and be able to have a field that is a List of some sort.

Let's say my .proto file looked like this:

   message SomeMessage
   {
      string datetime = 1;
      message inputData // This would be a list
      {
         repeated Object object = 1;
      }
      message Object 
      {
          ? // this need to be of a generic type - This is my question
          // My work around - Using extentions with some Object
          //List all primitive scalar types as optional and create an extension 100 to    max;
      }
      message someObject //some random entity - for example, employee/company etc.
      {  
          optional string name = 1; optional int32 id = 2;
      }
      extend Object 
      {
          optional someObject obj = 101;
      }
  } 

And this would be fine, and would work, and I'd have a List where Objects could be of any primitive type or could be List < someObject >. However- The problem here, is that any time I needed to handle a new type of object, I'd need to edit my .proto file, recompile for C# and java (The languages I need it for)...

If protocol buffers is not able to handle generic object types, is there another alternative that can? Any help on this matter is greatly appreciated.

Subsoil

The proto language supports neither generics nor inheritance. That is not supported AFAIK. – Danube

Thank you Marc! I will put this as an answer. – Subsoil

Check out this latest doc to know about supported datatypes : developers.google.com/protocol-buffers/docs/proto3#json – Cantrip

Answer 1

As Marc Gravell stated above - Protocol Buffers do not handle generics or inheritance.

Subsoil

Answer 2

Though I am late, just for the sake of new audience, you can use bytes in place of object and that can be any object which you can serialize/de-serialize.

Halliday

ok, we can map it like bytes, but how we can handle it on client side?? – Exocentric

Client is the one storing it. You can have a fully qualified class name as one of the protobuf property – Halliday

Answer 3

Here is the protobuf 3 definition of Struct which basically uses oneof to define such "generic" message type.

Birdwatcher

Answer 4

It IS possible to achieve generic message functionality but still adding new types will require rebuilding proto classes.

You use wrapper class

message Wrapper {
    extensions 1000 to max;
    required uint32 type = 1;
}

Then add some types

message Foo {
    extend Wrapper {
        optional Foo item = 1000;
    }

    optional int attr1_of_foo = 1;
    optional int attr2_of_foo = 2;
    optional int attr3_of_foo = 3;
}

message Bar {
    extend Wrapper {
        optional Bar item = 1001;
    }

    optional int attr1_of_bar = 1;
    optional int attr2_of_bar = 2;
    optional int attr3_of_bar = 3;
}

See how we extending Wrapper class in classes that we want to be stored by Wrapper class using extension.

Now, example of creating Foo wrapped object. I'm using Python, since it's most condensed form. Other languages can do the same.

wrapper = Wrapper()
wrapper.type = Foo.ITEM_FIELD_NUMBER
foo = wrapper.Extensions[Foo.item]
foo.attr1_of_foo = 1
foo.attr2_of_foo = 2
foo.attr3_of_foo = 3
data = wrapper.SerializeToString()

And example of deserializing

wrapper = Wrapper()
wrapper.ParseFromString(data)
if wrapper.type == Foo.ITEM_FIELD_NUMBER:
    foo = wrapper.Extensions[Foo.item]
elif wrapper.type == Bar.ITEM_FIELD_NUMBER:
    bar = wrapper.Extensions[Bar.item]
else:
    raise Exception('Unrecognized wrapped type: %s' % wrapper.type)

Now, because you want generic collection, make Wrapper a repeated field of other message and voilà.

Of course it's not complete solution, this architecture will need some more packaging to make it easy to use. For more information read about Protobuf extensions, especially nested ones (https://developers.google.com/protocol-buffers/docs/proto#nested) or google about item marshalling.

Interlocutress

Answer 5

If needed for java isn't this step be just for proper documentation sakes since performing a call on the PyObject wrapper have the ability to pass parameters to the object itself(?)

Landgrave

I find your post hard to read (you could try to improve readability by using punctuation, to make more obviously separated complete sentences). But I think you are asking a question which you want an answer to. Am I right? – Deadman