How to run an etcd cluster among pod replicas?

Question

I have a pod/service running an application that consumes etcd as a synchronization system and datastore. I want to run etcd within the pod, such that all of the replicas form a coherent cluster. In other words, so the application in replica #1 can write "foo" to localhost:4001/v2/keys/my_key and then replica #2 can then read localhost:4001/v2/keys/my_key and get "foo" as a result.

It's not clear how this can be done, since pod replicas are not individually addressable. I could in theory create an "etcd" service exposing the cluster ports, but any requests would round-robin to all the replicas so the individual etcd nodes would not be able to find each other.

Am I approaching this problem the correct way?

Calv

Answer 1

You can deploy etcd on kubernetes using an Operator (from the extensions/v1beta1) and the quay.io/coreos/etcd-operator image.

An example deployment with a cluster size of 3 looks like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  replicas: 1
  template:
    metadata:
      name: etcd-operator
      labels:
        app: etcd
        component: operator
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.3.0
        env:
        - name: MY_POD_NAMESPACE
          valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
        - name: MY_POD_NAME
          valueFrom: { fieldRef: { fieldPath: metadata.name } }

---

apiVersion: etcd.coreos.com/v1beta1
kind: Cluster
metadata:
  name: etcd-cluster
  labels:
    app: etcd
    component: cluster
spec:
  size: 3
  version: "3.1.8"

Please be aware of the beta status of this project. However according to the maintainers the operator is now stable. I have deployed the configuration above successfully but I didn't run any of this in production.

The operator code is available on github. You can find additional documentation there.

Lambertson

You are using apiVersion: etcd.coreos.com/v1beta1 and kind: Cluster, whereas etcd-operator example uses apiVersion: etcd.database.coreos.com/v1beta2 and kind: EtcdCluster. What are the differences? github.com/coreos/etcd-operator/blob/master/example/… – Fromm

Answer 2

There's a pretty good example of a three node etcd cluster here: https://github.com/coreos/etcd/tree/master/hack/kubernetes-deploy

They make use of separate rc's and services for each replica as a workaround until nominal services are added.

Laurinda

CAUTION: This hack is completely unsuitable for production since you CANNOT remove a node/replica and then add one again. There is no service discovery in this setup! – Cleave

Answer 3

I added your question to kubernetes/kubernetes#5017

If someone knows the answer, they will hopefully post it there.

I think it may require the "nominal services" feature (kubernetes/kubernetes#260) which is not implemented yet, but I'm not sure.

Israelisraeli