Kubernetes Ingress non-root path 404 Not Found

Question

I have the following that config that works when I try <NodeIP>:30080

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        name: app-node
    spec:
      containers:
        - name: app
          image: myregistry.net/repo/app:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          env:
            - name: NODE_ENV
              value: production
---
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    name: app-node
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
      nodePort: 30080
  type: NodePort

I am trying to use an Ingress:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
   name: nginx-ingress
 spec:
   rules:
   - host: myhost.com
     http:
       paths:
       - path: /app
         backend:
           serviceName: app-service
           servicePort: 80

myhost.com works with the nginx intro screen, but myhost.com/app gives 404 Not Found. Where is the issue in my setup?

---

UPDATE:

   - path: /
     backend:
       serviceName: app-service
       servicePort: 80

If I do root path it works, but how come /app doesn't?

Inadvisable

root(/) might be pointing to default backend which would be nginx welcome page, but /app might not be configured properly that,s why when routing request to that gives 404. try other url e.g myhost.com/app1 and see if nginx page shows up. – Plasty

Answer 1

Your ingress definition creates rules that proxy traffic from the {path} to the {backend.serviceName}{path}. In your case, I believe the reason it's not working is that /app is proxied to app-service:80/app but you're intending on serving traffic at the / root. Try adding this annotation to your ingress resource: nginx.ingress.kubernetes.io/rewrite-target: /

Source: https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/rewrite

Deplume

If I understand correctly, yes I would like myhost.com/app/foo => app-service:80/foo, not myhost.com/app/foo => app-service:80/app/foo. That explains why my config is wrong, and why only the root config (path: /) works, however nginx.ingress.kubernetes.io/rewrite-target: / does not seem to point it correctly. I have also tried ingress.kubernetes.io/rewrite-target: / with no luck. – Inadvisable

How did you install nginx-ingress? Do you have manifests or instructions? – Deplume

console.bluemix.net/docs/containers/cs_ingress.html#ingress (Step 3: Create the Ingress resource) – Inadvisable

Ah indeed. That annotation is specific to nginx, but your ingress controller is managed by IBM (according to the docs). If you were installing a custom nginx-ingress controller that annotation would probably work. In your case, check out the annotation options. Seems like rewrite-path should do it instead of nginx.ingress.kubernetes.io/rewrite-target – Deplume

I tried ingress.bluemix.net/rewrite-path: / as well, but it didn't work, and doesn't seem to function the same as rewrite-target. Confusing. Perhaps I should just consult their support, as this issue seems specific to them as a provider. I appreciate your help in pointing me towards the right direction though, thanks! – Inadvisable

One last shot -- did you try this? console.bluemix.net/docs/containers/… – Deplume

It finally works! Thank you! I had to do two things: 1. ingress.bluemix.net/rewrite-path: "serviceName=app-service rewrite=/" AND 2. path: /app/ (missing trailing slash!!!). – Inadvisable

@Deplume you're a life savior. Works like charm. – Woodpile

In my case (on mac with docker desktop), it worked by changing annotations to the following. (in ingress yaml) annotations: kubernetes.io/ingress.class: nginx – Kernel

Answer 2

As specified by brandon-barnett the issue is coming from the path but after reading the link he shared, I realised that a more specific rewrite rule would have to be specified for it to work. In my case

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - path: /something(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: http-svc
            port: 
              number: 80

The trick is with /$2 at annotation and (/|$)(.*) at the path. So what this rewrite does is that it picks everything that comes after something/ and replaces $2 with it, so:

• rewrite.bar.com/something rewrites to http-svc:80/
• rewrite.bar.com/something/ rewrites to http-svc:80/
• rewrite.bar.com/something/new rewrites to http-svc:80/new
• rewrite.bar.com/something/new/old rewrites to http-svc:80/new/old

Abaxial

Answer 3

In case it's useful to others:

I had the same thing, and it was nothing to do with Kubernetes Ingress Nginx. It was the Nginx server in my pod running the UI (Vue.JS in my case).

Solution:

1. Add an nginx.conf file to my UI app.

    user  nginx;
    worker_processes  auto;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    events {
        worker_connections  1024;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        sendfile        on;
        keepalive_timeout  65;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        access_log  /var/log/nginx/access.log  main;
        server {
            listen       80;
            root /usr/share/nginx/html;
            index index.html;
            location / {
                try_files $uri $uri/ /index.html;
            }
        }
    }
   

2. Add this line to my Dockerfile for the Vue.js app to copy the nginx.conf to your container.

    # production stage
    FROM nginx:stable-alpine as production-stage
    COPY nginx.conf /etc/nginx/
    COPY --from=build-stage /app/dist /usr/share/nginx/html
    EXPOSE 80
    CMD ["nginx", "-g", "daemon off;"]
   

Ref: https://mcmap.net/q/471211/-how-to-config-nginx-for-vue-router-on-docker

Bluebonnet

Same here! But in my case ReactJs running in kubernetes. Forgot to add custom nginx.conf to my deployment – Annulment

Answer 4

I ran into similar problem, but I have set PATHBASE in k8s yml, found out my NET8 Api didn't use the pathbase setting

In Program.cs add this

app.UsePathBase(builder.Configuration["PATHBASE"] ?? "/");

and yml like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapi-deployment
spec:
  selector:
    matchLabels:
      app: myapi
  template:
    metadata:
      labels: 
metadata:
  labels:
    app: myapi
spec:
  containers:
  - name: myapi
    image: my.azurecr.io/myapi
    env:
    - name: "ASPNETCORE_ENVIRONMENT"
      value: "PROD"
    - name: PATHBASE  <------------
      value: "/api"  <------------
    volumeMounts:
    - name: azure-fileshare
      mountPath: /app/Uploads    

Brunhild

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Universal

Answer 5

Other answers are already solving the issue, but there is another way without the annotations part in the yaml file. You have 2 ways to use ingress in Kube: with path or with host.

Imagine we have 2 services: api and dashboard.
With path, you would use myapp.example.fr/api and myapp.example.fr/dashboard.
With host, you would use api.myapp.example.fr and dashboard.myapp.example.fr

The answers above explained how to do with path, here is how you can do with host.
In your ingress you can set the path to / and your api routes will stay the same:

- path: /
  pathType: Prefix

However, this can be a bad solution if you have several services. In this case, you can use multiple hosts:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  ingressClassName: nginx
  rules:
    - host: api.myapp.example.fr  <-- first host
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: api-service
              port:
                number: 8080
    - host: dashboard.myapp.example.fr.  <-- second host
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: dashboard-service
              port:
                number: 8501

You can then access your 2 or more services through each dns.

Note that if you use Streamlit you will need to use / path or add streamlit run dashboard.py --server.baseUrlPath=<dashboard_path_here> when you run the dashboard.

Also note that you should not forget to deploy the nginx ingress controller (easy to do with helm).

Forlini