Where to store proto files which are shared among projects?

Question

I have project A and project B. They may be in different programming languages. Project A exposes an API using proto files, which project B will use to then generate the API in the programming language which project B uses.

But where are the proto files stored? What is the conventional way to do it with protobuf? Do you add the files generated from the proto files files to version control?

If you store a copy of the proto files in both project A and project B, then if project A changes its API, then project B will have to copy them over. This approach doesn't work great when there are many project using the API exposed by project A.

You can solve the above issue if you have a separate project, project C, containing the shared proto files. But then how do you generate the proto files from project A and project B?

Coffin

Answer 1

I would suggest storing the .proto files in a separate project. These are the contract between your two projects, and they are not necessarily "owned" by either one. Storing them in a separate project provides neutral ground for both project members to negotiate changes to the files - for example through a pull/merge request process where there may be members from both projects acting as reviewers.

As for generating code from the proto files, I would probably do this in the projects that need them. So in your case project C would only contain the .proto files, and projects A and B would pull the .proto files in and generate the code that they need. I feel like it has to be this way since it is projects A and B that are consuming the protobuf generated code. If the code was generated in project C then projects A and B would still have to pull the generated code to be able to use it, and since project C is technically decoupled from A and B it would not be obvious which languages would need to be generated - all of them? Just the 2 that are needed?

By creating project C you are creating a place that could potentially hold more .proto files for other projects. Thinking to the future you may have many projects that share common base message types. To manage an architecture with many interconnected projects it makes a lot of sense to try and consolidate message definitions, and this would be difficult / impossible if each project maintained it's own definitions, even worse (as you say) if there are duplicate copies. Storing them in one location allows for new projects to pick up existing definitions and expand them (within evolutionary guidelines), and allows for more rigor over managing and maintaining the set of definitions, e.g. a set of experienced reviewers making sure that everything is being done consistently and sensibly - be it from a modeling, namespacing or versioning perspective.

Hodges

So this is great and definitely in the line of https://mcmap.net/q/450081/-how-to-maintain-proto-files but how does project a or b “pull in” just a proto file from an external repo and use it to compile the code? Maybe you can look at my comments to that answer. Trying to figure this out – Brittnee

One way would be by adding project C as a git submodule to project A/B. Another would be to publish the files to a location that is accessible to both. Another still would be to curl the files direct from the source project. My preference would be to use git submodules. – Hodges

Should gRPC stubs be in project C or A/B? ex. .pb.go files – Voracity

@Voracity The answer suggests that the gRPC stubs should go into the consuming projects (A & B), but if you only use Go for consuming applications then they could go into project C. – Hodges

@Hodges if you have multiple consumers in the same language, and store the stubs in the individual projects, won't there be duplicate code? or is the a tradeoff we're fine having? – Voracity

@COOKIE I think that's the challenge here. The choice is: duplication of proto files, duplication of stubs, or extra complexity. I think there are other situational factors that would influence this, so there's no "right" answer, and the answer from Yehuda Makarov offers a great "everything in project C" approach. – Hodges

Answer 2

I will offer a small deviation from @JGC ‘s great answer. See https://www.bugsnag.com/blog/libraries-for-grpc-services for more details (for the gist of the approach rather than an imperative comparison).

When you put your proto files in a separate repository, that very repository can have generated client code as well. With a client in Golang, for instance, the generated code (which would also be Golang) can be imported even though it is in a separate repository. Meaning project a and/or b can easily import that generated code from project c.

It can be that for different languages, importing generated client code from project c may require more than just having a file sitting in the repo. But I can imagine that project c can have different ci/cd approaches set up to allow proper packages to be published.

Imagine one proto in project c is used to generate a go file, and that can be imported in another go project (project a). And project c also publishes a generated JavaScript file (or whatever) to an npm registry. I don’t know how dart works enough yet but imagine it also generated the client code for your flutter app and you grabbed it from project c as well.

See this question How to maintain proto files for more information.

Brittnee

Answer 3

A third alternative to the above two answers is to opt for a monorepo pattern. It's not ideal to have multi-language monorepos, but it does have an advantage where an interface such as proto files needs to be shared.

The advantage is that you can ensure more easily during both development and CI that there are no breaking changes.

The disadvantage is, of course, the complexity of setting a monorepo up and all the other complexities this entails.

Monkhmer