Kubernetes: No Route to Host

Question

I have a Bare-Metal Kubernetes custom setup (manually setup cluster using Kubernetes the Hard Way). Everything seems to work, but I cannot access services externally.

I can get the list of services when curl:

https://<ip-addr>/api/v1/namespaces/kube-system/services

However, when I try to proxy (using kubectl proxy, and also by using the <master-ip-address>:<port>):

https://<ip-addr>/api/v1/namespaces/kube-system/services/toned-gecko-grafana:80/proxy/

I get:

Error: 'dial tcp 10.44.0.16:3000: connect: no route to host'
Trying to reach: 'http://10.44.0.16:3000/'

• Even if I normally curl http://10.44.0.16:3000/ I get the same error. This is the result whether I curl from inside the VM where Kubernetes is installed. Was able to resolve this, check below.

• I can access my services externally using NodePort.

• I can access my services if I expose them through Nginx-Ingress.

• I am using Weave as CNI, and the logs were normal except a couple of log-lines at the beginning about it not being able to access Namespaces (RBAC error). Though logs were fine after that.

• Using CoreDNS, logs look normal. APIServer and Kubelet logs look normal. Kubernetes-Events look normal, too.

• Additional Note: The DNS Service-IP I assigned is 10.3.0.10, and the service IP range is: 10.3.0.0/24, and POD Network is 10.2.0.0/16. I am not sure what 10.44.x.x is or where is it coming from.

  • Also, I am using Nginx-Ingress (Helm Chart: https://github.com/helm/charts/tree/master/stable/nginx-ingress)

Here is output from one of the services:

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "kubernetes-dashboard",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
    "uid": "5c8bb34f-c6a2-11e8-84a7-00163cb4ceeb",
    "resourceVersion": "7054",
    "creationTimestamp": "2018-10-03T00:22:07Z",
    "labels": {
      "addonmanager.kubernetes.io/mode": "Reconcile",
      "k8s-app": "kubernetes-dashboard",
      "kubernetes.io/cluster-service": "true"
    },
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"addonmanager.kubernetes.io/mode\":\"Reconcile\",\"k8s-app\":\"kubernetes-dashboard\",\"kubernetes.io/cluster-service\":\"true\"},\"name\":\"kubernetes-dashboard\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":8443}],\"selector\":{\"k8s-app\":\"kubernetes-dashboard\"}}}\n"
    }
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 443,
        "targetPort": 8443,
        "nodePort": 30033
      }
    ],
    "selector": {
      "k8s-app": "kubernetes-dashboard"
    },
    "clusterIP": "10.3.0.30",
    "type": "NodePort",
    "sessionAffinity": "None",
    "externalTrafficPolicy": "Cluster"
  },
  "status": {
    "loadBalancer": {

    }
  }
}

I am not sure how to debug this, even some pointers to the right direction would help. If anything else is required, please let me know.

---

Output from kubectl get svc:

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns-primary        ClusterIP   10.3.0.10    <none>        53/UDP,53/TCP,9153/TCP   4h51m
kubernetes-dashboard   NodePort    10.3.0.30    <none>        443:30033/TCP            4h51m

---

EDIT:

Turns out I didn't have kube-dns service running for some reason, despite having CoreDNS running. It was as mentioned here: https://github.com/kubernetes/kubeadm/issues/1056#issuecomment-413235119

Now I can curl from inside the VM successfully, but the proxy-access still gives me the same error: No route to host. I am not sure why or how would this fix the issue, since I don't see DNS being in play here, but it fixed the issue regardles. Would appreciate any possible explanation on this too.

Affectionate

When you proxy do you mean using kubectl proxy? or just the proxy endpoint. Where are you running your curl requests from? Can you also post some output for kubectl get svc – Buggery

By proxy, I meant kubectl proxy and https://<master-id-addr>:<port>/api/v1/namespaces/kube-system/services/toned-gecko-grafana:80/proxy/ Curl request as run inside the VM on which the Kubernetes is installed. I have posted the output of kubectl get svc above in question. – Affectionate

Did going through kubectl proxy work before and not it doesn't work? – Buggery

No, it never worked. I just setup my cluster a few hours ago.... Also, I updated the question, please check the final part/edit. – Affectionate

Also, post your kubectl proxy command – Buggery

kubectl proxy --port=9001 --address='<ip-addr>' --accept-hosts="^*$". Here <ip-addr> is the address of Worker-server on which K8s Dashboard and other serivces are hosted (that's what I am trying to access). – Affectionate

Can you just do kubectl proxy only to talk to the apiserver? It should pick up the config under ~/.kube/config – Buggery

Yes, that works. I was able to access API endpoints-list and get service-names from server. EDIT: However, when I run curl localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ from within VM, I get No route to Host. – Affectionate

So everything works fine except contacting services externally. I am able to hit API-Server and get the response. But as soon as I access any service via proxy such as localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/, I get the No route to host. – Affectionate

I don't know if you can... – Buggery

It does say in the docs that you should be able to: kubernetes.io/docs/tasks/access-application-cluster/… Unless I am missing something... – Affectionate

Oh yeah, the dashboard. tbh not really sure what's going on. – Buggery

I see, thanks for the attempt anyway. Hopefully someone else can help. – Affectionate

Answer 1

I encountered the same issue and resolved it by running the commands below:

iptables --flush
iptables -tnat --flush
systemctl stop firewalld
systemctl disable firewalld
systemctl restart docker

Rescript

FYI this disables the firewall and is not a good option for security – Unpeg

Just flushing iptables (so first two commands) was enough in my case. – Misshapen

Answer 2

for me the the solution was to modify the rules in iptables as described here

sudo iptables -D  INPUT -j REJECT --reject-with icmp-host-prohibited
sudo iptables -D  FORWARD -j REJECT --reject-with icmp-host-prohibited

Interstellar

Answer 3

Try to run curl https://<master-ip-address>:<port>.

If port is open you should receive a message related to certificate or HTTPS.

It port is closed (which is probably the issue in your case) - the no route to host message appears.

(!) Do not disable iptables.

Open the port by SSH-ing into master node and running:
A ) sudo firewall-cmd --add-port=<relevant-port>/tcp --permanent (I'm assuming firewalld is installed).

B ) sudo firewall-cmd --reload.

C ) sudo firewall-cmd --list-all - you should see <relevant-port> is updated.

public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client ssh
  ports: <relevant-port>/tcp <---- Here
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Hourly

FYI: Ubuntu-based hosts don't have firewall-cmd installed. You can try ufw to manage the iptables, but it creates its own set of rules that might conflict with the existing iptables rules. – Ratable

Answer 4

When you are using kubectl proxy, by default you should use 127.0.0.1:8001 as HTTP Kube API URL. Your requests to http://127.0.0.1:8001 are then augmented with authentication headers and passed to API server. Thus you should try http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/toned-gecko-grafana:80/proxy/ rather then with https and api ip

Also, make sure you have socat installed on kube nodes.

Fallingout