Correct format of protoc go_package?

Question

I have an existing project in Go where I'm using Protocol buffers / gRPC. Until recent the go_package option was optional and the resulting Go package name would be the same as the proto package name.

This file lives in the project root. The generated code file (authenticator.pb.go) is living in the same location. Proto file:

syntax = "proto3";

package authenticator;

service Authenticator {...}

Generation command specifies I want to output in the same directory:

protoc --go_out=plugins=grpc:. authenticator.proto

Today I've pulled new version of the protocol buffers compiler and github.com/golang/protobuf/protoc-gen-go. Upon the first run a got a warning:

WARNING: Missing 'go_package' option in "authenticator.proto",
please specify it with the full Go package path as
a future release of protoc-gen-go will require this be specified.
See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

The suggested link is more or less useless. But the tutorial is a bit more explicit:

The go_package option defines the import path of the package which will contain all the generated code for this file. The Go package name will be the last path component of the import path. For example, our example will use a package name of "tutorialpb".

option go_package = "github.com/protocolbuffers/protobuf/examples/go/tutorialpb";

After adding this option to the proto file and rerunning the command, the output ends up in this path, relative to the project root. Something like:

$GOPATH/src/github.com/<org>/authenticator/github.com/<org>/authenticator/authenticator.pb.go

I've tried the following alternatives as go_package names:

• .
• authenticator

Generation happened in the correct location, but I got the warning back:

WARNING: Deprecated use of 'go_package' option without a full import path...

So what is the correct way without breaking the project layout?

Offutt

Answer 1

For pure proto generations you can do the following

protoc --go_out=paths=source_relative:./gen -I. authenticator.proto

or to generate for grpc the following can be used

protoc --go_out=plugins=grpc:./gen --go_opt=paths=source_relative authenticator.proto

If that fails you likely have a newer version of protoc and need to use the following command instead.

protoc --go-grpc_out=./gen --go-grpc_opt=paths=source_relative authenticator.proto

This is just Google making isolated decisions once again about how you should write and manage your code. However if you dig through enough issues of people having similar problems I found the above command can now be used. In addition to adding the full import path as shown in the other answer

option go_package = "github.com/example/path/gen;gen";

Following the above package and protoc command you would have your proto file in the project root with a module name of github.com/example/path. Your code would then be placed inside the gen folder which you will be responsible for creating.

You may need to adjust the output location of source_relative for your use but at least you don't end up with the duplication of paths or having to rely on placing your code in GOPATH again.

This may break going forward because of the additional changes found in this ticket. Unfortunately the binary name is exactly the same so if you run into issues read that ticket and try switching the version that you have installed until the new protoc-gen-go is ready for prod.

Aggappora

Is there a way to change the module path via protoc? I'm looking to generate Go code for both a client and server that are part of different modules, I tried using go_opt=module but it doesn't work with source_relative. – Furiya

It looks like the plugins part in --go_out=plugins=grpc is no longer supported. I was able to run the following for a similar result: protoc --go-grpc_out=./gen --go-grpc_opt=paths=source_relative authenticator.proto – Lima

It seems to me one needs multiple such options, i.e. --go_opt=paths=source_relative --go-grpc_opt=paths=source_relative --validate_opt=paths=source_relative – Grimmett

Answer 2

You just need to add a Go option which will tell to add file relative to proto file.

protoc --go_out=. --go_opt=paths=source_relative your_file.proto 

This will make the your_file.pb.go in same directory as your_file.proto

Registered

Answer 3

Hello I will explain step by step how I have generated it by reading the documentation

Create folder grpcgo

mkdir grpcgo && cd grpcgo

Create project

    go mod init projectxxx
    ## Download and install dependencies
    go get google.golang.org/protobuf/cmd/protoc-gen-go
    go get github.com/golang/protobuf

   sudo snap install protobuf or sudo apt  install protobuf-compiler

Create folder proto and inner folder the file .proto

wishlist.proto

syntax = "proto3";

package grpcgo; #Folder

option go_package = "pb;gen"; <-- Add 

message Item {
    enum ItemPriority {
        LOW = 0;
        MID = 50;
        HIGH = 100;
    }

    enum ItemStatus {
        INACTIVE = 0;
        ACTIVE = 1;
    }

    string id = 1;
    string wishListId = 2;
    string name = 3; 
    string link = 4;
    double price = 5;

    ItemPriority priority = 6;
    ItemStatus status = 7;

}

message WishList {
    enum WishListStatus {
        INACTIVE = 0;
        ACTIVE = 1;
    }

    string id = 1;
    string name = 2;
    WishListStatus status = 3;

}

Compiling proto

protoc -I=$PWD --go_out=$PWD $PWD/proto/*.proto

Documentation:

https://developers.google.com/protocol-buffers/docs/gotutorial

https://blog.friendsofgo.tech/posts/introduccion-a-grpc/

Version:

github.com/golang/protobuf v1.5.2

google.golang.org/protobuf v1.26.0

go version go1.16.5 linux/amd64

#EDIT

$ protoc -I=$PWD --go_out=$PWD $PWD/proto/*.proto
protoc-gen-go: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go_out: protoc-gen-go: Plugin failed with status code 1.

is solution is instaall protoc-gen-go

sudo snap install protobuf               # version 3.14.0, or
sudo apt  install golang-goprotobuf-dev  # version 1.3.4-2
sudo apt  install protoc-gen-go          # version 1.27.1-1

Edit 2

create dir pb in the root project

protoc -I=$PWD --go_out=$PWD/pb $PWD/protos/*.proto

syntax = "proto3";

package grpcgo;

option go_package = "../pb"; <--

message Item {
    enum ItemPriority {
        LOW = 0;
        MID = 50;
        HIGH = 100;
    }

    enum ItemStatus {
        INACTIVE = 0;
        ACTIVE = 1;
    }

    string id = 1;
    string wishListId = 2;
    string name = 3; 
    string link = 4;
    double price = 5;

    ItemPriority priority = 6;
    ItemStatus status = 7;

}

message WishList {
    enum WishListStatus {
        INACTIVE = 0;
        ACTIVE = 1;
    }

    string id = 1;
    string name = 2;
    WishListStatus status = 3;

}

versions:
protoc-gen-go v1.27.1
protoc        v3.12.4

hug from Argentina

Attar

Answer 4

The abovementioned error disappeared when I added

option go_package = "github.com/monkrus/grpc-from0;grpc_from0"; as below:

Olfe

This fixes the error but still leaves you with a more than undesirable file placement for existing projects. – Aggappora

The option go_package was already mentioned on the question. – Offutt

Answer 5

If you are using syntax = "proto3";. This issue will bound to happen after the change in grpc code generation processing.To fix this issue

You can follow the doc https://protobuf.dev/reference/go/go-generated/#package

step 1: paste this line in your your_file_with.proto

 option go_package = "pb;gen";
   

step 2 : Suppose you want to generate all files in pb folder replce the pb with your sutable name then run cmd in your terminal window.In my case my all protofile is in proto folder in project directory

 protoc -I=proto --go_out=. proto/*.proto

Shifra

Answer 6

I'm using Go and I want to create *.pb.go in the same folder with *.proto what I do is in *.proto

option go_package = "./{Your folder name}";

Take note, Your folder name will be reflected in the generated *.pb.go package. Then run:

protoc --go_out=.. *proto

Ammann

Answer 7

Although the documentation doesn't show, I've made it work by setting the output directory to GOPATH/src.

protoc --go_out=plugins=grpc:$(go env GOPATH)/src authenticator.proto 

Hope this can be a time saver for anybody else.

Offutt

This just looks like a hack where you are once again forced to place all of your go code in a special location. – Aggappora

@mschuett; yes it was a hack indeed. More a workaround in order to continue work until someone came up with a better solution. – Offutt

Answer 8

If your package name is user_service

user_service.proto definition like this:

syntax = "proto3";

package user_service;

service UserService {
}

just add option go_package = "./;user_service"; under package user_service; like this :

syntax = "proto3";

package user_service;

option go_package = "./;user_service";

service UserService {
}

Run the following: protoc *.proto --go_out=plugins=grpc:./ within the user_serivce directory. This should work.

Anallese

Answer 9

You can simply define your optional import path as follows

option go_package = ".;<Your_Import_path>";

For example, we can assume the package path as "/pub", So the statement as follows.

option go_package = ".;pub";

Then you can simply execute the protoc command to generate the pb.go file

protoc -I=<ABS_PATH_OUTPUT_DIR> --go_out=<ABS_PATH_PROTO_FILE>

Disjunction

Answer 10

Per testing in the go-grpc-http-rest-microservice-tutorial project https://medium.com/@amsokol.com/tutorial-how-to-develop-go-grpc-microservice-with-http-rest-endpoint-middleware-kubernetes-af1fff81aeb2

and source code https://github.com/amsokol/go-grpc-http-rest-microservice-tutorial/tree/part2

Trying to generate protobufs but the command kept failing. Running the command by itself could cause the command to fail silently, or if I tried running the command prefixed with sudo I may get the error:

Please specify a program using absolute path or make sure the program is available in your PATH system variable --go_out: protoc-gen-go: Plugin failed with status code 1.

I knew the path wasn't a problem, I could echo $PATH and see the GOBIN and GOROOT paths were in path (albeit I am not explicitly exporting either of those vars directly), i.e. at the bottom of .bashrc

# export GOPATH=$HOME/go
# export GOBIN=$HOME/go/bin
# export GOROOT=/usr/local/go/bin
# export PATH=$PATH:$GOBIN:$GOROOT
export PATH=$PATH:$HOME/go/bin:/usr/local/go/bin


$ echo $PATH
/home/tngai/.local/bin:/usr/local/lib/npm/bin:/usr/local/sbin:/usr/local/bin:/usr        /sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/include:/home/tngai/go/bin:/usr/local/go/bin

and if I typed in protoc-gen-go it would not error (but appear to take me to another command line, which I ctrl+c to exit from), so that was not a problem

Long story short, I noted that two params are involved for determining the output path to the generated files and also that If you make the path too long, it may error silently.

Here is it working: command to generate, where the first two are just file paths to find the proto file and it's dependency proto files, and the go_out which is the prefix path for the output file(s):

//cmd to run to generate protobuf and add it to the output directory
protoc --proto_path=api/proto/v1 --proto_path=third_party --go_out=plugins=grpc:./output todo-service.proto

// or below to add it under the pkg directory
protoc --proto_path=api/proto/v1 --proto_path=third_party --go_out=plugins=grpc:./ todo-service.proto

when combined to the path mentioned in the proto file itself from the required option go_package, which is the next part of the output path.

//todo-service.proto file
syntax = "proto3";
package v1; //name of the folder this is in
 
import "google/protobuf/timestamp.proto";
import "google/api/annotations.proto"; 
//import "protoc-gen-swagger/options/annotations.proto";

//new required line I had to add, and which also determines output path of gen file(s)
option go_package = "./pkg/api/v1";

//...commented out the swagger generation per testing just the protobuf generation...

// Task we have to do
message ToDo {
// Unique integer identifier of the todo task
int64 id = 1;
// ...

Ankylostomiasis

Answer 11

Add this to your .proto source file

option go_package = "relative_path_to_go_package"; // i.e: ./ if the .proto source file is in the root folder

syntax = "proto3";
package chat;

option go_package = "./";

message Message {
    string body = 1;
}

service ChatService {
    rpc SayHello(Message) returns (Message) {}
}

Spinney