ingress points to wrong port on a service

Question

I have a Kubernetes service that exposes two ports as follows

Name:              m-svc
Namespace:         m-ns
Labels:            
Annotations:       <none>
Selector:          app=my-application
Type:              ClusterIP
IP:                10.233.43.40
Port:              first  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.233.115.178:8080,10.233.122.166:8080
Port:              second  8888/TCP
TargetPort:        8888/TCP
Endpoints:         10.233.115.178:8888,10.233.122.166:8888
Session Affinity:  None
Events:            <none>

And here is the ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: f5
    virtual-server.f5.com/http-port: "80"
    virtual-server.f5.com/ip: controller-default
    virtual-server.f5.com/round-robin: round-robin
  creationTimestamp: 2018-10-05T18:54:45Z
  generation: 2
  name: m-ingress
  namespace: m-ns
  resourceVersion: "39557812"
  selfLink: /apis/extensions/v1beta1/namespaces/m-ns
  uid: 20241db9-c8d0-11e8-9fac-0050568d4d4a
spec:

  rules:
  - host: www.myhost.com
    http:
      paths:
      - backend:
          serviceName: m-svc
          servicePort: 8080
        path: /first/path
      - backend:
          serviceName: m-svc
          servicePort: 8080
        path: /second/path
status:
  loadBalancer:
    ingress:
    - ip: 172.31.74.89

But when I go to www.myhost.com/first/path I end up at the service that is listening on port 8888 of m-svc. What might be going on?

Another piece of information is that I am sharing a service between two ingresses that point to different ports on the same service, is this a problem? There is a different ingress port the port 8888 on this service which works fine

Also I am using an F5 controller

After a lot of time investigating this, it looks like the root cause is in the F5s, it looks like because the name of the backend (Kubernetes service) is the same, it only creates one entry in the pool and routes the requests to this backend and the one port that gets registered in the F5 policy. Is there a fix for this? A workaround is to create a unique service for each port but I dont want to make this change , is this possible at the F5 level?

Isthmian

Could you provide some steps to recreate the issue? I could check if this occurs in my lab environment. – Kelm

Deploy an app with two exposed ports , create a single service with two exposed ports as in the question and then create a path ingress as in the question – Isthmian

Answer 1

From what I see you don't have a Selector field in your service. Without it, it will not forward to any backend or pod. What makes you think that it's going to port 8888? What's strange is that you have Endpoints in your service. Did you manually create them?

The service would have to be something like this:

Name:              m-svc
Namespace:         m-ns
Labels:            
Annotations:       <none>
Selector:          app=my-application
Type:              ClusterIP
IP:                10.233.43.40
Port:              first  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.233.115.178:8080,10.233.122.166:8080
Port:              second  8888/TCP
TargetPort:        8888/TCP
Endpoints:         10.233.115.178:8888,10.233.122.166:8888
Session Affinity:  None
Events:            <none>

Then in your deployment definition:

selector:
  matchLabels:
    app: my-application

Or in a pod:

apiVersion: v1
kind: Pod
metadata:
  annotations: { ... }
  labels:                                
    app: my-application

You should also be able to describe your Endpoints:

$ kubectl describe endpoints m-svc
Name:         m-svc
Namespace:    default
Labels:       app=my-application
Annotations:  <none>
Subsets:
  Addresses:          x.x.x.x
  NotReadyAddresses:  <none>
  Ports:
    Name    Port  Protocol
    ----    ----  --------
    first   8080  TCP
    second  8081  TCP

Events:  <none>

Edgeways

Thanks, it exhibits the same result with selectors unfortunately and i did not manually set the endooints – Isthmian

I think it is pointing to port 8888 because when i hit the ingress host from the browser it shows me the service that’s listening on port 8888 and not 8080 – Isthmian

Do I need a selector field in the ingress definition? – Isthmian

Not really, you don't. Btw, I tried this on my setup and it works for me with an nginx ingress controller. – Edgeways

Is your f5 load balancer a layer 4 or layer 7 load balancer? – Edgeways

I created a service for each port, this is an F5 limitation that it only creates an entry once for the service so the second port is compromised – Isthmian

Answer 2

Your Service appears to be what is called a headless service: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services. This would explain why the Endpoints was created automatically.

Something is amiss because it should be impossible for your HTTP request to arrive at you pods without the .spec.selector populated.

I suggest deleting the Service you created and delete the Endpoints with the same name and then recreate the Service with type=ClusterIP and the spec.selector properly populated.

Gardie

Hi Ive update the question to have a selector which doesnt seem to fix the problem, what do you mean by endpoints with the same name? – Isthmian