Kubernetes service : randomly getting Connection refused

Question

When curl is made inside pod on port 80, response is fine. Calling curl outside container via Kubernetes service on machines IP and port 30803, sporadically "Connection refused" appears.

nginx app config:

server {
        listen  80;
        server_name 127.0.0.1;
        access_log  /var/log/nginx/access.log;
        error_log   /var/log/nginx/error.log;
        root        /usr/share/nginx/html;
        index index.html;

       error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

Kubernetes deployments and service manifest which is used:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: dev
  labels:
    environment: dev
spec:
  selector:
      matchLabels:
         environment: dev
  replicas: 1
  template:
    metadata:
      labels:
        environment: dev
    spec:
      containers:
      - name: web-app
        imagePullPolicy: Never
        image: web-app:$BUILD_ID
        ports:
          - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-dev-svc
  namespace: dev
  labels:
    environment: dev
spec:
  selector:
    environment: dev
  type: NodePort
  ports:
  - name: http
    nodePort: 30803
    port: 80
    protocol: TCP
    targetPort: 80

Belter

You should do some verification with curl (-v flag). Also, do a kubectl get pods to ensure the pod is not restarting and is in a ready state. You can also do a kubectl get services to also ensure that only one service is using that nodeport. This is kind of the first step for debugging. You can also later on look at the logs in your k8s node. You will probably find something about network issue in there. – Rattler

Did all the debug steps, describe/logs pod . Pod is running perfectly, since is just doing reverse proxy. I'm thinking that nginx image version is not compatible with some k8s features, or to try to switch to another webserver, in order to see what may be the reason of such behavior ... – Belter

NGinx works great even if you don't have a real Ingress controller. The nodeport is used only if you have no SSL certificate for a domain. Otherwise all is quite easy to setup – Rattler

Yes, used everywhere nginx as webserver/reverse-proxy, but it's odd why via K8s services such, behavior. I assume something on K8s level …. – Belter

You are using label environment: dev. Is there anything else deployed in the dev namespace with the same label that could be selected by the service? kubectl get po --show-labels – Mccaskill

Yes!!! Thank you Matt . Removed all the deployments and services which are in dev namespaces. Now works! Will work to make labels right! – Belter

Would you like to provide the Answer for better visibility, explaining what the issue was and how you solved it? I'd be happy to upvote your answer. – Mccaskill

The issue was that 2 services in selector was using same label value - 'environment: dev' , and I assume this random connection was provoked, because it was balancing between one pod to another. Fixed labels values, now works perfectly. – Belter

Answer 1

The issue was that 2 services in selector was using same label value - 'environment: dev' , and I assume this random connection was provoked, because it was balancing between one pod to another. Fixed labels values, now works perfectly.

Mccaskill

Answer 2

When I run K8s with a NodePort, I don't have any problem. You can try first by using a proxy (port-forward) to your service and then your pod to ensure that all is working with the same behavior. If doing the port-forward to your pod directly works without any issue, then you might have an issue between your service and pod (e.g.: network policies such as too many calls in a short amount of time).

Regarding my nginx config, it's quite simple:

# /usr/share/nginx/html # cat /etc/nginx/conf.d/default.conf
server {
    listen       80;

    # Optional
    listen  [::]:80;
    server_name  localhost;

    # Default configuration
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;

        # Forward everything to the react router
        try_files $uri $uri/ /index.html?$args;
    }

    # If we wish some custom error page later, we could also add them.
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Else, if you want to try with a verbatim nginx, simply use the nginx image (see kubernetes.io Cheat Sheet)

kubectl create deployment nginx --image=nginx
kubectl create services nodeport nginx-svc --tcp 30666:80 

# Nodeport should then be 30666 on the cluster

Rattler

Thanks for response. The problem was in Kubernetes Labels. Same value for labels was used for multiple apps – Belter