How to connect to Bigtable Emulator from a GoLang application? How to use it?

Question

I am trying to use BigTable Emulator. I have never used it before. I followed the documentation but not able to understand,

1. How to connect an application to Emulator.
2. How to set BIGTABLE_EMULATOR_HOST environment variable.

Please help by explaining it or pointing me to any examples or links. Thank you.

Classicist

Answer 1

The BIGTABLE_EMULATOR_HOST environment variable overrides the normal connection logic. Having it set causes a client to connect to the emulator without any special changes needed in the code (i.e. the project/instance values passed to NewAdminClient()/NewClient() are ignored).

The command provided in step 2 of the Using the emulator instructions sets the environment variable for you.

See the implementation of GoogleCloudPlatform/golang-samples/bigtable/helloworld/main.go for a simple, concrete example of interacting with a BigTable instance.

---

Walkthrough

To demonstrate using the Bigtable Emulator from go, I'm using a free tier, f1-micro Compute Engine instance. It's an instance where I hadn't used go before, so it's basically a clean slate.

First things first, I setup my go environment:

$ sudo apt install golang
<... SNIP apt output ...>
$ mkdir ~/go
$ export GOPATH=$HOME/go

Then I updated the google-cloud-sdk package and installed the google-cloud-sdk-bigtable-emulator package:

$ sudo apt install google-cloud-sdk
<... SNIP apt output ...>
$ sudo apt install google-cloud-sdk-bigtable-emulator
<... SNIP apt output ...>

At this point the emulator is ready to run, so I started it using the documented command:

$ gcloud beta emulators bigtable start
Executing: /usr/lib/google-cloud-sdk/platform/bigtable-emulator/cbtemulator --host=localhost --port=8086
[bigtable] Cloud Bigtable emulator running on 127.0.0.1:8086

After starting it, it runs in the foreground, so I left that terminal alone and started a new one.

In the new terminal, the first thing is setting the BIGTABLE_EMULATOR_HOST environment variable. This step is documented in the emulator instructions. The following command generates the shell command for setting the variable. To see the command it generates it, run it directly:

$ gcloud beta emulators bigtable env-init
export BIGTABLE_EMULATOR_HOST=localhost:8086

Running the command enclosed in $(...) as documented tells the shell to execute the output of the command, thereby setting the variable:

$ $(gcloud beta emulators bigtable env-init)

Now that the emulator is running and the environment is ready, I need a client application to to connect to it. I chose to use the helloworld application from the GoogleCloudPlatform/golang-samples repository.

$ git clone https://github.com/GoogleCloudPlatform/golang-samples.git
$ cd golang-samples/bigtable/helloworld

In the main.go file in the above directory, the cloud.google.com/go/bigtable and golang.org/x/net/context packages are imported, so I ran go get to fetch both of them.

$ go get cloud.google.com/go/bigtable
$ go get golang.org/x/net/context

Then I ran the example application, using dummy values for the -project and -instance flags, since it's going to be connecting to the locally running emulator.

$ go run main.go -project foo -instance bar
2018/06/18 00:39:27 Creating table Hello-Bigtable
2018/06/18 00:39:27 Writing greeting rows to table
2018/06/18 00:39:27 Getting a single greeting by row key:
2018/06/18 00:39:27     greeting0 = Hello World!
2018/06/18 00:39:27 Reading all greeting rows:
2018/06/18 00:39:27     greeting0 = Hello World!
2018/06/18 00:39:27     greeting1 = Hello Cloud Bigtable!
2018/06/18 00:39:27     greeting2 = Hello golang!
2018/06/18 00:39:27 Deleting the table

Nothing really happens in the emulator terminal to indicate it's being used. So, to demonstrate that it's actually interacting with the emulator, I stopped the emulator process (by pressing Ctrl-C in it's terminal) and tried executing the client application again:

$ go run main.go -project foo -instance bar
2018/06/18 00:40:03 Retryable error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:8086: getsockopt: connection refused", retrying in 47.77941ms
2018/06/18 00:40:03 Retryable error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:8086: getsockopt: connection refused", retrying in 2.153551ms
2018/06/18 00:40:03 Retryable error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp [::1]:8086: getsockopt: connection refused", retrying in 114.145821ms
<... SNIP repeated errors ...>
^Csignal: interrupt
$

So, using an unmodified client application interacts with the emulator just by having the proper environment variable set.

Willed

Thank you, It worked. I am using Mac, so just the Go installation steps are different for me. :) – Classicist

The example works for me as well; however, I cannot seem to get it working in a docker compose environment. I have the bigtable container running within docker compose but my service container cannot connect to it. I made sure to set the environment variable in my service container for BIGTABLE_EMULATOR_HOST but no luck. Any ideas? – Bloodworth

I'm not familiar with the Docker Compose environment, but you may need to set BIGTABLE_EMULATOR_HOST to a value other than localhost:8006 in order to target an emulator instance running on a different container. If you continue to have issues, it would be best to ask a new question. – Willed