Dataflow pipeline and pubsub emulator

Question

I'm trying to setup my development environment. Instead of using google cloud pubsub in production, I've been using the pubsub emulator for development and testing. To achieve this I set the following environment variable:

export PUBSUB_EMULATOR_HOST=localhost:8586

This worked for the python google pubsub library but when I switched to using java apache beam for google dataflow, the pipeline still points to production google pubsub. Is there a setting, environment variable or method on the pipeline that I need to set so that the pipeline reads for the local pubsub emulator?

Penstock

Answer 1

I found the solution in the PubsubOptions interface, and extending it for my own PipelineOptions implementation. Then using setPubsubRootUrl() to set it to localhost:port of the emulator.

Penstock

Ah, It was so simple. stupid me. PubsubOptions options = PipelineOptionsFactory.fromArgs(args).withValidation() .as(PubsubOptions.class); options.setPubsubRootUrl("localhost:8085"); – Untruth

Answer 2

You can also specify this option in a configuration file

pubsubRootUrl=http://localhost:8185

or via program argument (haven't tested but should also work)

-Dexec.args="--pubsubRootUrl=http://localhost:8185"

the http:// is required to conform to Java URL format

Crigger