How do I generate python grpc code from within a setuptools installer (setup.py)?

Question

We have some proto files for gRPC in a repo and I read that it is not good to commit generated code. So I figured I need to have the generation as part of the package installation (e.g. setuptools, setup.py)

However, to generate gRPC code, you need to first install the package by running pip install grpcio-tools according to the docs. But the purpose of setup.py is to automatically pull down dependencies like grpcio-tools.

So is there a best-practice for doing this? As in, how to generate code that depends on another python package from within setuptools? Am I better off just create a separate build.sh script that manually pip-installs and generates the code? Or should I expect users of the package to already have grpcio-tools installed?

Celle

Good question. You can generate those files when the package is being built; any packages you need for that can be passed via setup_requires argument. – Hausfrau

Answer 1

As far as I know, the "current" best practice is:

• pip manages dependencies
• setup.py performs build

Executing "pip install ." is almost equivalent to perform "pip install -r requirements.txt" + "python setup.py build" + "python setup.py install".

This is a custom command that generates python sources from proto files:

class GrpcTool (Command):
    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def run(self):
        import grpc_tools.protoc

        proto_include = pkg_resources.resource_filename('grpc_tools', '_proto')

        grpc_tools.protoc.main([
            'grpc_tools.protoc',
            '-I{}'.format(proto_include),
            '--python_out=SOME_PATH/',
            '--grpc_python_out=SOME_PATH/',
            'SOME_PROTO.proto'
        ])

that is invoked customizing build_py command, like this:

class BuildPyCommand (build_py):
    def run(self):
        self.run_command('grpc')
        super(BuildPyCommand, self).run()

Note the import inside the run method. It seems that pip run setup.py several times, both before and after having installed requirements. So if you have the import on top of file, the build fails.

Froh

I wish there was a concrete example of this. I'm struggling to find a way, and none of this crap is documented in any reasonable way. – Precedence

Answer 2

Along with @makeroo approach, alternative way is to execute grpc_tools module as a subprocess.

The benefit of this approach is to receive a generation result for sure; 0 is a success and 1 for error.

proto_files = ["proto/file1.proto", "proto/file2.proto"]

import subprocess
for file in proto_files:
    args = "--proto_path=. --python_out=. --grpc_python_out=. {0}".format(file)
    result = subprocess.call("python -m grpc_tools.protoc " + args, shell=True)
    print("grpc generation result for '{0}': code {1}".format(file, result))

Above code will create generated python files to proto directory where .proto files reside.

Dutch