Kubernetes kubeadm init fails due to dial tcp 127.0.0.1:10248: connect: connection refused

Question

I'm trying to setup a very simple 2 node k8s 1.13.3 cluster in a vSphere private cloud. The VMs are running Ubuntu 18.04. Firewalls are turned off for testing purposes. yet the initialization is failing due to a refused connection. Is there something else that could be causing this other than ports being blocked? I'm new to k8s and am trying to wrap my head around all of this.

I've placed a vsphere.conf in /etc/kubernetes/ as shown in this gist. https://gist.github.com/spstratis/0395073ac3ba6dc24349582b43894a77

I've also created a config file to point to when I run kubeadm init. Here's the example of it'\s content. https://gist.github.com/spstratis/086f08a1a4033138a0c42f80aef5ab40

When I run sudo kubeadm init --config /etc/kubernetes/kubeadminitmaster.yaml it times out with the following error.

[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

Checking sudo systemctl status kubelet shows me that the kubelet is running. I have the firewall on my master VM turned off for now for testing puposes so that I can verify the cluster will bootstrap itself.

   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Sat 2019-02-16 18:09:58 UTC; 24s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 16471 (kubelet)
    Tasks: 18 (limit: 4704)
   CGroup: /system.slice/kubelet.service
           └─16471 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-config=/etc/kubernetes/vsphere.conf --cloud-provider=vsphere --cgroup-driver=systemd --network-plugin=cni --pod-i

Here are some additional logs below showing that the connection to https://192.168.0.12:6443/ is refused. All of this seems to be causing kubelet to fail and prevent the init process from finishing.

    Feb 16 18:10:22 k8s-master-1 kubelet[16471]: E0216 18:10:22.633721   16471 kubelet.go:2266] node "k8s-master-1" not found
    Feb 16 18:10:22 k8s-master-1 kubelet[16471]: E0216 18:10:22.668213   16471 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://192.168.0.12:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-master-1&limit=500&resourceVersion=0: dial tcp 192.168.0.1
Feb 16 18:10:22 k8s-master-1 kubelet[16471]: E0216 18:10:22.669283   16471 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://192.168.0.12:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.0.12:6443: connect: connection refused
    Feb 16 18:10:22 k8s-master-1 kubelet[16471]: E0216 18:10:22.670479   16471 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.0.12:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-master-1&limit=500&resourceVersion=0: dial tcp 192.1
    Feb 16 18:10:22 k8s-master-1 kubelet[16471]: E0216 18:10:22.734005   16471 kubelet.go:2266] node "k8s-master-1" not found

Monahon

Answer 1

In order to address the error (dial tcp 127.0.0.1:10248: connect: connection refused.), run the following:

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo kubeadm reset
sudo kubeadm init

Use the same commands if the same error occurs while configuring worker node.

Spue

Can you offer additional explanation of what these commands do? That will help future readers understand how to solve similar (but not identical) problems. It will also help readers feel more comfortable executing these on their system, if they first understand what they do. – Monoxide

I think it is documented around here kubernetes.io/docs/setup/production-environment/… – Isisiskenderun

@Monahon Did you check if swap if disabled or not? I had the same issue and later realized swap was on and had to disable it to get it running. :) – Kept

@SantoshBitra Disabling the Swap it's works for me. Thanks, man – Reincarnate

Answer 2

You cannot use bootstrap-kubeconfig to initialize the master's kubelet since -- as you are experiencing -- it has no api server to contact in order to generate its private key and certificate. Catch-22. I am about 80% certain that removing the --bootstrap-kubeconfig from the kubelet args will help that situation. I would expect that the kubelet already has its key and cert in /var/lib/kubelet/pki, so that might be worth checking, too.

Also, assuming you are using the /etc/kubernetes/manifests directory to run the apiserver and controllermanager, ensure that staticPodPath: in /var/lib/kubelet/config.yaml points to the right directory. It's unlikely to be the problem, but it's super cheap to check.

Hustings

Thanks for your answer, I verified that kubelet's cert and key are in fact in the PKI folder. I'm honestly not sure what you mean when you say I can't use bootstrap-kubeconfig though. At this point all I have done is placed the config files I mentioned in their folders and ran the kubeadm init command on the master. Was there more I was supposed to do before that? I'm really new to k8s and was following a guide. – Monahon

to follow up, I also checked /var/lib/kubelet/config.yaml and noticed it didn't contain a staticPodPath property. – Monahon

why I can't use bootstrap-kubeconfig though because that mechanism is designed for (plus or minus) bare metal deployments, such as using an autoscaling group in AWS (or its GCP equivalent) -- with just kubelet and kubeadm binaries, you can take an empty machine and have it join the cluster by contacting the API to get the cluster config, and then using the bootstrap token to auth long enough to request a certificate pair from the API. That's the "bootstrap-kubeconfig" part: auth using a token and then generate the real kubelet.conf from it. But, no API server, no bootstrap process – Hustings

So should I remove --config /etc/kubernetes/kubeadminitmaster.yaml from my init command? – Monahon

It seems that kubelet isn't able to start any of the containers that it needs. – Monahon

So I ran sudo kubeadm reset && sudo kubeadm init –pod-network-cidr=10.244.0.0/16 and the initialization worked fine. Would I be able to now set up my cloud provider configuration? – Monahon

Sure, but your question is turning into something waaay different from "connection refused to :10248" so I would suggest opening a new question for your new problem, because there is no way a solution will fit in these comments – Hustings

I guess looking back at your original answer, how would I actually remove --bootstrap-kubeconfig from the kubelet args? – Monahon

vi /lib/systemd/system/kubelet.service and/or vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (I don't know which of those files contains the offending argument). Given your extreme lack of interest in debugging your own problem, are you sure you wouldn't be happier with a more plug-and-play solution like kubespray? – Hustings

I've been trying to debug the issues but what you were telling me was far over my head since I'm really new to linux, k8s, containers and vSphere. Really short time frame to get something up and running and have been trying to follow docs and guides but have obviously been struggling...Appreciate all the help you have tried to provide though :) – Monahon

Answer 3

Try adding the below into vi /etc/docker/daemon.json

{ "exec-opts": ["native.cgroupdriver=systemd"] }

and then a systemctl restart docker.

Also, yes, daemon.json in /etc/docker would be something new that you'd create and it doesn't exist there by default.. :)

Kept

just a reminder again: Check if swap if disabled or not? I had the same issue and later realized swap was on and had to disable it to get it running. :) – Kept