Why unable to access a service if setting externalTrafficPolicy to Local in a kubernetes cluster

Question

I'm following this guide to apply the source ip feature to my kubernetes cluster.

Firstly, I created a pod by running:

$ kubectl run source-ip-app --image=gcr.io/google_containers/echoserver:1.4

Then expose it as a NodePort service:

kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort

At this point, I'm able to access the service from outside of the cluster and get correct client_address:

$ curl 10.74.68.49:16860 | grep client
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 296 0 296 0 0 43167 0 --:--:-- --:--:-- --:--:-- 49333
client_address=10.168.193.130

But if applying the source ip feature:

kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'

I'll get timeout:

$ curl 10.74.68.49:16860 | grep client
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:01:14 --:--:-- 0curl: (7) Failed to connect to 10.74.68.49 port 16860: Operation timed out

I'm wondering what's the reason behind this and how to resolve it.

My env info:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T07:00:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T06:43:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Update:

My cluster has 2 nodes, I get the timeout issue no matter which node ip is accessed.

Delagarza

tutorial docs say that packets will be dropped and not forwarded if there is no local endpoint for the service. Can you post output of kubectl get ep nodeport -o json | grep nodeName – Orangewood

@stacksonstacks, thanks for the comment. The endpoints got are: "nodeName": "10.74.68.48", "nodeName": "10.74.68.49". The cluster is deployed with --hostname-override=${ip}. According to this, the kube-proxy can't find local endpoints due to the rename. I'm trying to figure out how to avoid this. Because in my prod env, the hostname is overridden. – Delagarza

That thread says the setting --hostname-override=${ip} on each kube-proxy should fix the issue. Have you done that and restarted the kube-proxy pods? – Orangewood

I'm attempting to do it, as my kube-proxy is a DaemonSet, I'm investigating how to pass --hostname-override to it. – Delagarza

Answer 1

Create kube-proxy.yaml

kubectl get ds -n kube-system kube-proxy -o yaml > kube-proxy.yaml

Edit kube-proxy.yaml to include the HOST_IP argument

# ...
spec:
  containers:
  - command:
    - ./hyperkube
    - proxy
    - --cluster-cidr=10.2.0.0/16
    - --hostname-override=$(HOST_IP)
    - --kubeconfig=/etc/kubernetes/kubeconfig
    - --proxy-mode=iptables
    env:
    - name: HOST_IP
      valueFrom:
          fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
    #...

Update the pods:

kubectl apply -f kube-proxy.yaml

This will apply the fix mentioned in https://github.com/kubernetes/kubernetes/issues/48437, resolving the dropped packets issue.

Orangewood