Login/Register
How to solve Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store
Asked Answered
kuberneteskubernetes-ingressnginx-ingress
C

3

7

I know this question has been asked before, but the previous one doesn't apply to my case. I have 2 deployments and 2 services and I am using ingress resource and nginx ingress controller

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: frontend-ingress
    annotations: 
      kubernetes.io/ingress.class: nginx
spec:
    rules:
    - http:
        paths:
        - path: /api
          backend:
            serviceName: frontend-svc
            servicePort: 3000
        - path: /
          backend:
            serviceName: static-svc
            servicePort: 80

And the 2 services are defined below,

services.yaml

  apiVersion: v1
  kind: Service
  metadata:
      labels:
          app: frontend
      name: frontend-svc
  spec:
      ports:
      - port: 3000
        protocol: TCP
        targetPort: 3000
      selector:
          app: frontend
      type: ClusterIP

---

apiVersion: v1
kind: Service
metadata:
    labels:
        app: static
    name: static-svc
spec:
    ports:
    - port: 80
      protocol: TCP
      targetPort: 80
    selector:
        app: static
    type: ClusterIP

The deployments are shown below,

apiVersion: apps/v1
kind: Deployment
metadata:
    labels:
        app: frontend
    name: frontend
spec:
    replicas: 2
    selector:
        matchLabels:
            app: frontend
    template:
        metadata:
            labels:
                app: frontend
        spec:
            containers:
            - image: magalixcorp/sample-api:v1
              imagePullPolicy: IfNotPresent
              name: frontend
              env:
              - name: REDIS_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: redis-password
                    key: redis-password
              volumeMounts:
              - name: config-volume
                mountPath: /app/config.json
                subPath: config.json
            volumes:
                - name: config-volume
                  configMap:
                    name: app-config

---

apiVersion: apps/v1
kind: Deployment
metadata:
    labels:
        app: static
    name: static
spec:
    replicas: 2
    selector:
        matchLabels:
            app: static
    template:
        metadata:
            labels:
                app: static
        spec:
            containers:
            - image: magalixcorp/static:v1
              imagePullPolicy: IfNotPresent
              name: static

I have deployed an nginx controller using helm to AKS. The routes are not working at all. When I check the nginx-controller logs i get this

W1109 16:30:33.892443       6 controller.go:849] Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store
W1109 16:30:33.892478       6 controller.go:849] Error obtaining Endpoints for Service "default/static-svc": no object matching key "default/static-svc" in local store
I1109 16:30:33.929583       6 main.go:112] "successfully validated configuration, accepting" ingress="frontend-ingress/default"
W1109 16:30:33.935904       6 controller.go:849] Error obtaining Endpoints for Service "default/frontend-svc": no object matching key "default/frontend-svc" in local store
W1109 16:30:33.935930       6 controller.go:849] Error obtaining Endpoints for Service "default/static-svc": no object matching key "default/static-svc" in local store
I1109 16:30:33.935973       6 controller.go:144] "Configuration changes detected, backend reload required"
I1109 16:30:33.936084       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-ingress", UID:"efde0161-c03f-49d4-8f2b-38e2ceaaf8a7", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"7945", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1109 16:30:34.005219       6 controller.go:161] "Backend successfully reloaded"
I1109 16:30:34.005615       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"nginx", Name:"nginx-ingress-nginx-controller-68f45f49f6-slnv7", UID:"7880b0be-a1ff-450d-b1c1-4cee11722a6b", APIVersion:"v1", ResourceVersion:"3538", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1109 16:30:37.269565       6 controller.go:144] "Configuration changes detected, backend reload required"
I1109 16:30:37.342929       6 controller.go:161] "Backend successfully reloaded"
I1109 16:30:37.343677       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"nginx", Name:"nginx-ingress-nginx-controller-68f45f49f6-slnv7", UID:"7880b0be-a1ff-450d-b1c1-4cee11722a6b", APIVersion:"v1", ResourceVersion:"3538", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1109 16:31:01.926983       6 status.go:290] "updating Ingress status" namespace="default" ingress="frontend-ingress" currentValue=[] newValue=[{IP: Hostname:a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com}]
I1109 16:31:01.937928       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-ingress", UID:"efde0161-c03f-49d4-8f2b-38e2ceaaf8a7", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"8081", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
10.0.1.165 - - [09/Nov/2020:16:32:18 +0000] "GET / HTTP/1.1" 200 1313 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 396 0.002 [default-static-svc-80] [] 10.0.3.144:80 1313 0.000 200 750627c05c4ca98a2bcd60ae8bbb8828
10.0.1.165 - - [09/Nov/2020:16:32:19 +0000] "GET /js/script.js HTTP/1.1" 200 314 "http://a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 397 0.001 [default-static-svc-80] [] 10.0.1.175:80 314 0.000 200 af72c4fdc493a489064307872636b662
10.0.1.165 - - [09/Nov/2020:16:32:20 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://a596c83cb03464004bcced0767dd5aa9-1681194742.us-east-1.elb.amazonaws.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 407 0.002 [default-static-svc-80] [] 10.0.3.144:80 153 0.000 404 ba36f92120a41720842ba95e24b9975c
10.0.2.31 - - [09/Nov/2020:16:32:47 +0000] "GET /.env HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" 231 0.001 [default-static-svc-80] [] 10.0.1.175:80 555 0.000 404 d8ec13faad1d46c13401b397e2d80ad0
10.0.2.31 - - [09/Nov/2020:16:32:48 +0000] "POST / HTTP/1.1" 405 559 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" 317 0.001 [default-static-svc-80] [] 10.0.1.175:80 559 0.000 405 567a4b2499fc816d72135445076bafda
10.0.1.165 - - [09/Nov/2020:16:32:53 +0000] "GET /api HTTP/1.1" 200 3 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 399 0.003 [default-frontend-svc-3000] [] 10.0.1.26:3000 3 0.000 200 8a92b9829d26dccb60f11a9c8537d831
10.0.3.231 - - [09/Nov/2020:16:34:37 +0000] "POST /api HTTP/1.1" 204 0 "-" "curl/7.64.1" 211 0.002 [default-frontend-svc-3000] [] 10.0.1.26:3000 0 0.000 204 cf837f8bc89c29f2d2b07ac317120220
10.0.2.31 - - [09/Nov/2020:16:34:55 +0000] "GET /api HTTP/1.1" 200 3 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0" 425 0.006 [default-frontend-svc-3000] [] 10.0.2.98:3000 3 0.004 200 a5757323ff32d8c4dbc56ef476798954

When I try :3000/api either to POST or GET it's not working. What could be wrong. What is it that am doing wrong.

Citarella answered 9/11, 2020 at 17:29 Comment(0)
W
4

Try cleaning this up a bit. So create a deployment just for your frontend. So this file you would call frontend-depl.yml and then write it like this:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: frontend-depl
spec:
    replicas: 1
    selector:
        matchLabels:
            app: frontend
    template:
        metadata:
            labels:
                app: frontend
        spec:
            containers:
              - name: frontend
                image: magalixcorp/sample-api:v1
---

apiVersion: v1
kind: Service
metadata:
    name: frontend-srv
spec:
    selector:
      app: frontend
    ports:
     - name: frontend
       protocol: TCP
       port: 3000
       targetPort: 3000

I am suggesting this to keep your yml files neat and tidy as this can all spin out of control real fast as you can tell and troubleshooting can be a pain.

Also I would go into your terminal and run:

$ kubectl get endpoints --namespace <namespace>
...
<namespace> <your-app-service>  <none> 21h

That is to look and see what endpoints you have.

The issue could also be your label selectors, which is why I offered a cleaner version of your yml file, at least the frontend-depl one, I would use the same pattern for your other yml files.

To learn more about label selectors: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors

Wes answered 9/11, 2020 at 18:51 Comment(0)
K
0

I had the same problem and it turned out that I was targeting the wrong IP address. I was sending the traffic from outside my VNet to the (external) IP address that I found here:

kubectl get ingress

I were confused because this address was working properly when I tested with a temporary running Pod that was just making a single HTTP call from within my VNet and then were terminating.

However, when I read the logs of the Ingress Pod I found a different IP address that was resolving to. Then I checked the endpoints by this command

kubectl get endpoints

And it turned out that the IP address that was listed here is the correct one that my Ingress controller is listening to. So, I configured my private DNS to use that IP address and resolved the problem.

Keffer answered 20/5, 2023 at 19:7 Comment(0)
M
0

The next commands can help you better diagnose the issue:

kubectl get endpoints

To get available endpoints based on your services

kubectl describe ingress

To get more information about the configuration of your ingress (can give you some logs)

Make sure that your ingress names and ports are correct. You can further debug the services with the next command:

kubectl port-forward svc/service-name 8080:{service-port}

And then access http://localhost:8080 to see if things are coming through.

I had a very similar error as you and the ports and names of my ingress were wrong from a previous "kubectl apply"

Maggoty answered 29/3 at 0:34 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.