I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.
My setup:
- ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record:
my-etcd-cluster.my-etcd-namespace.svc.cluster.local) - I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
- I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD
Here's my affected Go code:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
DialTimeout: 5 * time.Second,
Username: username,
Password: password,
})
if err != nil {
return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}
And here's an error I'm getting when clientv3.New(...) gets executed:
{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09 INFO controller.etcdclient Finish reconcile loop for some-service/test-svc-client {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09 ERROR controller.etcdclient Reconciler error {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
The same error happens when I'm passing some dummy, invalid credentials.
However, when I tried to access the database in a HTTP API manner:
postBody, _ := json.Marshal(map[string]string{
"name": username,
"password": password,
})
responseBody := bytes.NewBuffer(postBody)
resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()
...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.
When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.
EDIT: I've also added a simple pod to try accessing my etcd db via etcdctl:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: my-controller-namespace
spec:
containers:
- name: etcdctl
image: bitnami/etcd
command:
- sleep
- infinity
When logged into the container via kubectl exec, I was able to access my db:
$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK
I guess the problem is somewhere in the SDK?
