protoc --go_opt=paths=source_relative vs --go-grpc_opt=paths=source_relative

Question

I am having a hard time figuring out protoc command and go plugin.

What is the different between:

protoc \
   # Directory where you want the compiler to write your Go output.
   --go_out=.
   # vs ?
   --go_opt=paths=source_relative
   # vs ?
   --go-grpc_out=.
   # vs ?
   --go-grpc_opt=paths=source_relative

If --go_opt generate

• <name>.pb.go file

and --go-grpc_opt generate

• <name>_grpc.pb.go file

why even have --go_out?

Can you shed some light on protoc - the doc do not say anything about --go-grpc_opt?

And and protoc -h do not even list go as an OUT_DIR?

Note: I install using this doc

Silvers

Answer 1

why even have --go_out?

So, the thing to understand here is that gRPC is not the same as Protocol Buffers, gRPC uses Protocol Buffers but there are other frameworks that are using them as well. So we need to generate both.

Now, in order to generate the Protocol buffer related code, you need to use --go_out as you mentioned. but for the gRPC code you need to use --go-grpc_out.

and --go-grpc_opt generate _grpc.pb.go file

No, --go-grpc_out does.

Can you shade some light on protoc - the doc do not stay anything about --go-grpc_opt?

Then, before the generation of code you can pass some options and that's what --go_opt and --go-grpc_opt are for. The first one passes options for Protobuf generation and the second one for gRPC generation. Options are pretty obscure, and there is not official list of all of them, but you used source_relative (which tell protoc to use relative paths) for path and there is also the module option (which help protoc know the go module name to generate in proper folder)

And and protoc -h do not even list go as an OUT_DIR?

And finally, protoc doesn't officially support Go as output, you need to install an external plugin and that's why protoc -h doesn't show --go_out. A related discussion can be found here.

Skeptical

Answer 2

The protoc compiler supports different flags or options and the flags you use on the command line decides where the generated go code will be placed.

The official docs for these flags(atleast for paths=source_relative and module=$PREFIX) are not very clear and can be hard.

paths=source_relative

Here is what official docs says

If the paths=source_relative flag is specified, the output file is placed in the same relative directory as the input file. For example, an input file protos/buzz.proto results in an output file at protos/buzz.pb.go.

The above statement might be confusing and hard to understand at first because it is not giving complete context and folder layout of how the files are placed on the disk.

Here is how I understand it, when this flag is used the compiler generates the go code in the directory specified by --go_out and ensures that directory tree structure of the generated go code files matches the directory tree structure of source proto files. The word source is important here because it can change based on the value of the --proto_path

Let's say we have the following directory structure

❯ tree
.
├── go.mod
├── go.sum
└── src
    └── protos
        ├── baz
        │   └── bar.proto
        └── foo.proto

4 directories, 4 files

Here is the general syntax of the protoc command

protoc --proto_path=<IMPORT_DIRECTORY> --go_out=. --go_opt=paths=source_relative <ONE_OR_MORE_SOURCE_PROTO_FILES>

Now lets consider the following examples

# 1

❯ protoc --proto_path=src/protos --go_out=. --go_opt=paths=source_relative foo.proto baz/bar.proto

❯ ls
baz       foo.pb.go go.mod    go.sum    src

~/development/personal/craftup/go-grpc/proto-buffers
❯ ls -R baz
bar.pb.go

In the above example we specified the import directory by setting --proto_path=src/protos which makes the actual path for the source proto files to foo.proto and baz/bar.proto, and hence the generated pb files were placed in the current directory (--go_out=.) as foo.pb.go and bar/baz.pb.go.

Now let's change the --proto_path in the above command to src and see what happnes.

**# 2**

❯ protoc --proto_path=src --go_out=. --go_opt=paths=source_relative protos/foo.proto protos/bar/baz.proto

❯ ls -R src
protos

src/protos:
baz       foo.proto

src/protos/baz:
bar.proto

This time a new protos directory was created and the generated go files were placed inside it, why ? Because this time when we changed the import path to --proto-path=src the path for source proto files changed to protos/foo.proto and protos/baz/bar.proto.

# 3

Now let's finally mix --go_out flag here as well in here and see what happens

❯ mkdir out
❯ protoc --proto_path=src --go_out=out --go_opt=paths=source_relative protos/foo.proto protos/bar/baz.proto

❯ ls -R out
protos

out/protos:
baz       foo.pb.go

out/protos/baz:
bar.pb.go

This is exactly similar to last example except that we provided a custom directory to hold the generated code.

module=$PREFIX

If the module=$PREFIX flag is specified, the output file is placed in a directory named after the Go package’s import path, but with the specified directory prefix removed from the output filename. For example, an input file protos/buzz.proto with a Go import path of example.com/project/protos/fizz and example.com/project specified as the module prefix results in an output file at protos/fizz/buzz.pb.go. Generating any Go packages outside the module path results in an error. This mode is useful for outputting generated files directly into a Go module.

Lets see this as well as in action, consider the following proto file at location src/baz/bar.proto with following contents

syntax = "proto3";

package bar;

option go_package = "github.com/rbhanot/proto-buffers-practise/bar";

message GreetBar {
    string name = 1;
}

❯ protoc --proto_path=src --go_out=. --go_opt=module=github.com/rbhanot/proto-buffers-practise protos/foo.proto protos/baz/bar.proto
❯ ls
bar    foo    go.mod go.sum src

~/development/personal/craftup/go-grpc/proto-buffers
❯ ls -R foo bar
bar:
bar.pb.go

foo:
foo.pb.go

Lets see what happened here, the compiler removed the module prefix(github.com/rbhanot/proto-buffers-practise) from the go_package specified in the proto file, then created the the package directories in the current directory(--go_out=.) named after the remaining path in go_package option which were foo and bar and dumped the code files inside it.

Now lets change go_package = "github.com/rbhanot/proto-buffers-practise/src"; and run the same command as above

❯ protoc --proto_path=src --go_out=out --go_opt=module=github.com/rbhanot/proto-buffers-practise protos/foo.proto protos/baz/bar.proto

❯ ls
go.mod go.sum out    src

~/development/personal/craftup/go-grpc/proto-buffers
❯ ls out/src
bar.pb.go foo.pb.go

This time we see that go code was generated inside the out/src directory.

I hope this makes things more clear (it atleast did for me) wrt to how these flags behave.

Ortolan