How can I connect to CockroachDB from outside the Kubernetes cluster?

Question

I've set up and deployed a Kubernetes stateful set containing three CockroachDB pods, as per docs. My ultimate objective is to query the database without requiring use of kubectl. My intermediate objective is to query the database without actually shelling into the database pod.

I forwarded a port from a pod to my local machine, and attempted to connect:

$ kubectl port-forward cockroachdb-0 26257
Forwarding from 127.0.0.1:26257 -> 26257
Forwarding from [::1]:26257 -> 26257

# later, after attempting to connect:
Handling connection for 26257
E0607 16:32:20.047098   80112 portforward.go:329] an error occurred forwarding 26257 -> 26257: error forwarding port 26257 to pod cockroachdb-0_mc-red, uid : exit status 1: 2017/06/07 04:32:19 socat[40115] E connect(5, AF=2 127.0.0.1:26257, 16): Connection refused


$ cockroach node ls --insecure --host localhost --port 26257
Error: unable to connect or connection lost.

Please check the address and credentials such as certificates (if attempting to
communicate with a secure cluster).

rpc error: code = Internal desc = transport is closing
Failed running "node"

Anyone manage to accomplish this?

Herisau

Answer 1

From inside the Kubernetes cluster, you can talk to the database by connecting the cockroachdb-public DNS name. In the docs, that corresponds to the example command:

kubectl run cockroachdb -it --image=cockroachdb/cockroach --rm --restart=Never -- sql --insecure --host=cockroachdb-public

While that command is using the CockroachDB image, any Postgres client driver you use should be able to connect to cockroachdb-public when running with the Kubernetes cluster.

Connecting to the database from outside of the Kubernetes cluster will require exposing the cockroachdb-public service. The details will depend somewhat on how your Kubernetes cluster was deployed, so I'd recommend checking out their docs on that: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#exposing-the-service

And in case you're curious, the reason forwarding port 26257 isn't working for you is because port forwarding from a pod only works if the process in the pod is listening on localhost, but the CockroachDB process in the statefulset configuration is set up to listen on the pod's hostname (as configured via the --host flag).

Sindhi

Thanks! Got it working now by pointing the ExternalIPs of the cockroachdb-public service to a master node with a static ip – Herisau

could you go into more detail, and post your config files? Do you use an ingress controller? – Sigismundo

The config file is the one used in the previously linked documentation - raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/… . You could presumably use an ingress controller, but on any major cloud provider you could also just change your service to type NodePort or type LoadBalancer as explained in the Kubernetes docs I linked to in my answer. – Sindhi