"kubectl exec" results in "error: unable to upgrade connection: Unauthorized"

Question

I tried kubectl exec on a k8s 1.6.4 RBAC-enabled cluster and the error returned was: error: unable to upgrade connection: Unauthorized. docker exec on the same container succeeds. Otherwise, kubectl is working. kubectl tunnels through an SSH connection but I don't think this is the issue.

kubelet authn is enabled but not authz. The docs say that authz is AlwaysAllow by default, so I have left it this way.

I have a feeling that it is similar to this issue. But the error message is a tad different.

Thanks in advance!

Verbose logs for the kubectl exec command:

I0614 16:50:11.003677   64104 round_trippers.go:398] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.003705   64104 round_trippers.go:398] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.6.4 (darwin/amd64) kubernetes/d6f4332" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.169474   64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169493   64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169497   64104 round_trippers.go:426]     Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169500   64104 round_trippers.go:426]     Content-Length: 12
I0614 16:50:11.169502   64104 round_trippers.go:426]     Content-Type: text/plain; charset=utf-8
I0614 16:50:11.169506   64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169509   64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169512   64104 round_trippers.go:426]     Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169545   64104 round_trippers.go:426]     Content-Length: 12
I0614 16:50:11.169548   64104 round_trippers.go:426]     Content-Type: text/plain; charset=utf-8
F0614 16:50:11.169635   64104 helpers.go:119] error: unable to upgrade connection: Unauthorized

Adventitious

can you give some more details on this 'kubectl tunnels' how are you creating the tunnel using ssh. – Iatrogenic

I forward localhost:6443 to the worker node like this ssh -L 6443:localhost:6443 worker1.kube. Then I config kubectl to connect to localhost:6443. – Adventitious

Answer 1

This is an RTFM moment... The solution was basically to follow all the steps on this page for authn, authz, or both.

I had omitted --kubelet-client-certificate and --kubelet-client-key which resulted in the error. Without these flags, kube-apiserver will fail to authenticate with kubelet when you do a kubectl exec.

My original attempt to configure authn was by reading the docs for the kubelet daemon (ie. not the one above). Hence the grave omission.

Adventitious

Where do you add these parameters? – Dianoetic

It is configured in kube-apiserver's manifest. The official documentation will explain exactly how it works. – Adventitious

Answer 2

In my case (while learning Kubernetes The Hard Way, I had to configure RBAC permissions to allow the Kubernetes API Server to access the Kubelet API on each worker node. I had created a ClusterRole and ClusterRoleBinding to access the Kubelet API

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver

References:

• https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/13-kube-apiserver-to-kubelet.md

Cyzicus

Answer 3

Run across this one on my minikube cluster version 1.12.3
If you're running a minikube cluster, upgrade minikube and it'll be fixed.

check your minikube version:

$ minikube update-check
CurrentVersion: v1.12.3
LatestVersion: v1.13.0

Minikube docs
Upgrade(Mac OS):

brew upgrade minikube

Vikiviking

Answer 4

Sometimes such an error might be caused by container state.

If you create a pod on heavily loaded node, wait until it's state becomes 'Running' (kubectl get pod pod-name -o 'jsonpath={.status.phase}'), and immediately attempt to exec something on it, you may see error: unable to upgrade connection: Unauthorized, because container is not fully ready yet.

Waiting second or two before exec may solve problem in this case.

Patrickpatrilateral