Invalid x509 certificate for kubernetes master

Question

I am trying reach my k8s master from my workstation. I can access the master from the LAN fine but not from my workstation. The error message is:

% kubectl --context=employee-context get pods
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.161.233.80, not 114.215.201.87

How can I do to add 114.215.201.87 to the certificate? Do I need to remove my old cluster ca.crt, recreate it, restart whole cluster and then resign client certificate? I have deployed my cluster with kubeadm and I am not sure how to do these steps manually.

Eyeshade

Answer 1

One option is to tell kubectl that you don't want the certificate to be validated. Obviously this brings up security issues but I guess you are only testing so here you go:

kubectl --insecure-skip-tls-verify --context=employee-context get pods

The better option is to fix the certificate. Easiest if you reinitialize the cluster by running kubeadm reset on all nodes including the master and then do

kubeadm init --apiserver-cert-extra-sans=114.215.201.87

It's also possible to fix that certificate without wiping everything, but that's a bit more tricky. Execute something like this on the master as root:

rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=10.161.233.80,114.215.201.87
docker rm `docker ps -q -f 'name=k8s_kube-apiserver*'`
systemctl restart kubelet

Emmettemmey

I think this is not a legit answer anymore because this alpha subcommand is not available anymore. – Babineaux

For kubeadm 1.15, just run command kubeadm alpha certs renew all – Builder

because kubeadm alpha subcommand is not available anymore, you can replace it with the following command kubeadm init phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=10.161.233.80,114.215.201.87 – Corrugate

Answer 2

This command for new kubernetes >=1.8:

rm /etc/kubernetes/pki/apiserver.*
kubeadm alpha phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=10.161.233.80,114.215.201.87
docker rm -f `docker ps -q -f 'name=k8s_kube-apiserver*'`
systemctl restart kubelet

Also whould be better to add dns name into --apiserver-cert-extra-sans for avoid issues like this in next time.

Poitiers

You are a god!! – Winterize

I'm getting Error: unknown flag: --apiserver-advertise-address. I think this alpha subcommand has changed a lot since then. – Babineaux

Thank you. For anyone having problems. Alpha is just the command for any experimental commands. When they are no longer experimental they get graduated and removed from alpha. kubeadm init phase certs all worked fine for me. – Bearce

Answer 3

For kubeadm v1.13.3

rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=114.215.201.87
docker rm -f `docker ps -q -f 'name=k8s_kube-apiserver*'`
systemctl restart kubelet

Vedetta

works also nicely with v1.14.3. One thing I might add: Looks like private IPs are automatically added to --apiserver-cert-extra-sans, so you'll only have to add additional public IPs there – Console

Works also nicely with v1.17.2. I needed --apiserver-cert-extra-sans=aaa,bbb,ccc,ddd,eee. – Garrison

Answer 4

If you used kubespray to provision your cluster then you need to add a 'floating ip' (in your case its '114.215.201.87'). This variable is called supplementary_addresses_in_ssl_keys in the group_vars/k8s-cluster/k8s-cluster.yml file. After updating it, just re-run your ansible-playbook -b -v -i inventory/<WHATEVER-YOU-NAMED-IT>/hosts.ini cluster.yml.

NOTE: you still have to remove all the apiserver certs (rm /etc/kubernetes/pki/apiserver.*) from each of your master nodes prior to running!

Stokes

Answer 5

Issue cause: Your configs at $HOME/.kube/ are present with your old IP address.

Try running,

rm $HOME/.kube/* -rf
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Infract

This has nothing to do about .kube directory. It's possible that the IP does not match, but it doesn't mean we need to delete it. – Henricks

Answer 6

For Kubernetes 1.12.2/CentOS 7.4 the sequence is as follows:

rm /etc/kubernetes/pki/apiserver.*
kubeadm alpha phase certs all --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=51.158.75.136
docker rm -f `docker ps -q -f 'name=k8s_kube-apiserver*'`
systemctl restart kubelet

Silt

Answer 7

Use the following command:

kubeadm init phase certs all

Legumin

Answer 8

For me when I was trying to accessing via root (after sudo -i) I got the error. I excited and with normal user it was working.

Matherly

Answer 9

For me the following helped:

1. rm -rf ~/.minikube
2. minikube delete
3. minikube start

Probably items no 2 and 3 would have been sufficient

Spathose