How to make Protobuf 3 fields mandatory?

Question

I'm writing my first API endpoint in GoLang using GRPC/proto-buffers. I'm rather new to GoLang. Below is the file I'm writing for my test case(s)

package my_package

import (
    "context"
    "testing"

    "github.com/stretchr/testify/require"

    "google.golang.org/protobuf/types/known/structpb"
    "github.com/MyTeam/myproject/cmd/eventstream/setup"
    v1handler "github.com/MyTeam/myproject/internal/handlers/myproject/v1"
    v1interface "github.com/MyTeam/myproject/proto/.gen/go/myteam/myproject/v1"
)

func TestEndpoint(t *testing.T) {
    conf := &setup.Config{}

    // Initialize our API handlers
    myhandler := v1handler.New(&v1handler.Config{})

    t.Run("Success", func(t *testing.T) {
        res, err := myhandler.Endpoint(context.Background(), &v1interface.EndpointRequest{
            A: "S",
            B: &structpb.Struct{
                Fields: map[string]*structpb.Value{
                    "T": &structpb.Value{
                        Kind: &structpb.Value_StringValue{
                            StringValue: "U",
                        },
                    },
                    "V": &structpb.Value{
                        Kind: &structpb.Value_StringValue{
                            StringValue: "W",
                        },
                    },
                },
            },
            C: &timestamppb.Timestamp{Seconds: 1590179525, Nanos: 0},
        })
        require.Nil(t, err)

        // Assert we got what we want.
        require.Equal(t, "Ok", res.Text)
    })


}

This is how the EndpointRequest object is defined in the v1.go file included above:

// An v1 interface Endpoint Request object.
message EndpointRequest {

  // a is something.
  string a = 1 [(validate.rules).string.min_len = 1];

  // b can be a complex object.
  google.protobuf.Struct b = 2;

  // c is a timestamp.
  google.protobuf.Timestamp c = 3;

}

The test-case above seems to work fine.

I put validation rule in place that effectively makes argument a mandatory because it requires that a is a string of at least one. So if you omit a, the endpoint returns a 400.

But now I want to ensure that the endpoint returns 400 if c or b are omitted. How can I do that? In Protobufs 3, they got rid of the required keyword. So how can I check if a non-string argument was passed in and react accordingly?

Unfreeze

Answer 1

The short version: you can't.

required was removed mostly because it made changes backwards incompatible. Attempting to re-implement it using validation options is not quite as drastic (changes are easier), but will run into shortcomings as you can see.

Instead, keep the validation out of the proto definition and move it into the application itself. Anytime you receive a message, you should be checking its contents anyway (this was also true when required was a thing). It is a rare case that the simple validation provided by options or required is sufficient.

Polypropylene

But in my application itself, how can I detect if b and c were passed in or not? – Unfreeze

The accessor (eg: GetB()) will return a *Struct. When it is not specified in the message, the return value will be nil. see Go proto3 details. Scalars are a problem as they are not pointers but the plain type so you cannot tell the difference between an int being unset vs being 0 (see github issue for details and ways around that). – Polypropylene

Which ones are scalars and which ones are not? Strings, integers, timestamps, structs? – Unfreeze

There's a list in the link from the previous comment, and a link to the full scalar types tables. – Polypropylene

Answer 2

Required fields were removed in proto3. Here is github issue where you can read detailed explanation why that was done. Here is excerpt:

We dropped required fields in proto3 because required fields are generally considered harmful and violating protobuf's compatibility semantics. The whole idea of using protobuf is that it allows you to add/remove fields from your protocol definition while still being fully forward/backward compatible with newer/older binaries. Required fields break this though. You can never safely add a required field to a .proto definition, nor can you safely remove an existing required field because both of these actions break wire compatibility

IMO, that was questionable decision and obviously i'm not alone, who's thinking that. Final decision should have been left to developer.

Suricate