Should we run a Consul container in every Pod?

Question

We run our stack on the Google Cloud Platform (hosted Kubernetes, GKE) and have a Consul cluster running outside of K8s (regular GCE instances).

Several services running in K8s use Consul, mostly for it's CP K/V Store and advanced locking, not so much for service discovery so far.

We recently ran into some issues with using the Consul service discovery from within K8s. Right now our apps talk directly to the Consul Servers to register and unregister services they provide.

This is not recommended best-practice, usually Consul clients (i.e. apps using Consul) should talk to the local Consul agent. In our setup there are no local Consul agents.

My Question: Should we run local Consul agents as sidekick containers in each pod?

IMHO this would be a huge waste of ressources, but it would match the Consul best-practies better.

I tried searching on Google, but all posts about Consul and Kubernetes talk about running Consul in K8s, which is not what I want to do.

Cosmogony

Running sidekicks has a severe downside in autoscaling (and as you stated resource waste). Why don't you want to move consul into k8s? – Stinkweed

Because accessing internal resources within K8s is still not really practical. Also Consul has state, which is pretty immature in K8s. – Cosmogony

I would like to state that different :) : State requires a stringent design pattern to be able to use this in any auto-scaling environment. Many stateful applications are just not implemented to be terminated at any point in time. – Stinkweed

why can't you run a single pod with consul agent in it and it exposed by a k8s service and all your app services configured to talk to this consul agent running as pod in the k8s cluster? – Mental

I agree with @Mental - why run in sidecar? Just run it in the cluster, in the same namespace and create a K8S service to access it. If you want to persist data mount a volume. Maybe look at Kelsey Hightower's GitHub repo on Kubernetes and Consul for some ideas. – Resourceful

github.com/kelseyhightower/consul-on-kubernetes – Quinta

Answer 1

As the official Consul Helm chart and the documentation suggests the standard approach is to run a DaemonSet of Consul clients and then use a connect-side-car injector to inject sidecars into your node simply by providing an annotation of the pod spec. This should handle all of the boilerplate and will be inline with best practices.

• Consul: Connect Sidecar; https://www.consul.io/docs/platform/k8s/connect.html

Glans