Traefik v2.2 Ingress Route example not working

Question

I am trying to setup traefik ingress route using the configuration that is provided at https://docs.traefik.io/routing/providers/kubernetes-crd/

I can see traefik is up & running, can also see the dashboard. But I dont see the whoami service on dashboard and cannot access it via url.

crd.yaml

# All resources definition must be declared
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

ingress.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: myingressroute
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik

  spec:
    entryPoints:
      - web

    routes:
      - match: Host(`test`) && PathPrefix(`/bar`)
        kind: Rule
        services:
        - name: whoami
          port: 80

rbac.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default

traefik.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.2
          args:
            - --log.level=DEBUG
            - --api
            - --api.insecure
            - --entrypoints.web.address=:80
            - --providers.kubernetescrd
          ports:
            - name: web
              containerPort: 80
            - name: admin
              containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: LoadBalancer
  selector:
    app: traefik
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 8080
      name: admin
      targetPort: 8080

whoami.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  namespace: default
  labels:
    app: containous
    name: whoami

spec:
  replicas: 1
  selector:
    matchLabels:
      app: containous
      task: whoami
  template:
    metadata:
      labels:
        app: containous
        task: whoami
    spec:
      containers:
        - name: containouswhoami
          image: containous/whoami
          ports:
            - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: default

spec:
  ports:
    - name: http
      port: 80
  selector:
    app: containous
    task: whoami

UPDATE After resources from link

E0708 21:34:10.222538       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1alpha1.IngressRouteUDP: ingressrouteudps.traefik.containo.us is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressrouteudps" in API group "traefik.containo.us" at the cluster scope
E0708 21:34:10.223416       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1alpha1.TLSStore: tlsstores.traefik.containo.us is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "tlsstores" in API group "traefik.containo.us" at the cluster scope
E0708 21:34:11.225368       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1alpha1.TLSStore: tlsstores.traefik.containo.us is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "tlsstores" in API group "traefik.containo.us" at the cluster scope

Maness

could you please share the deployment manifests to understand the exact issue. This info is not helping much. :) – Cohin

I updated the question with configuration files used. I am using minikube on mac , start it up and load these files. I don't do any ip or port changes, just updated /etc/hosts file to include minikube ip for test domain. – Maness

The error seems to be the rbac issue. Check the latest CRD on traefik official site and update. That should work. – Cohin

I have the same issue on AWS in an EKS cluster. I get a 404 when I curl the URL. – Scuba

Updated the configuration is shared in the answers. Please check and let me know if that works for you. – Cohin

Answer 1

Its working for me with with treafik 2.1 using following set of resources.

link to example

Can you try these once and let me know if this helps.

Merlinmerlina

Hello, thank you so much for sharing the link, it is very helpful. Configuration files listed on link work fine with traefik v2.1, but not 2.2. I am not sure why. I have updated some screenshots after running resources from this link. – Maness

Ahh okay. I better update my answer then :) . Anyway can you tell the error which you are getting ? like the treafik logs in Debug mode – Merlinmerlina

Answer 2

I deployed the below code and the whoami is now accessible without any issues. Things I changed are, updated the CRD, RBAC with the latest available in Traefik and changed the apiVersion for the deployment to "apps/v1". Simply copy the below code all together and deploy on kubernetes. Once it is up access http://localhost/whoami-app-api.

Deployment File:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.1
          args:
            - --accesslog=true
            - --api
            - --api.insecure
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            - --providers.kubernetescrd
            - --configfile=/config/traefik.toml
          ports:
            - name: web
              containerPort: 80
            - name: admin
              containerPort: 8080
            - name: websecure
              containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: LoadBalancer
  selector:
    app: traefik
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 443
      name: websecure
      targetPort: 80
    - protocol: TCP
      port: 8080
      name: admin
      targetPort: 8080
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-whoami
  namespace: default
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: PathPrefix(`/whoami-app-api`)
      kind: Rule
      services:
        - name: whoami
          port: 80

Cohin

The question is about v2.2. I see this deployment uses v2.1. I will try it. – Scuba

You have also referenced a toml file which you have not included. I have tried the current code and found that it still doesn't work. The example code does not use a toml file. I don't know if it makes any difference but the example code does not include the SSL port. – Scuba

was trying to replicate the code which they had mentioned above and didn't notice the image version. The toml file can be ignored as we have parsed all required config as arguments. I have updated image to 2.2 and tried, it works good for me. I'm suspecting the IngressRoute may be the issue for you. What does kubectl get ingressroute give for you. – Cohin

Additionally, if you deploying this on the EKS cluster, please check if the service created for traefik is in ready state. kubectl get svc. May be that is in Pending state. – Cohin

In my case, all nodes are ready, all pods are running and no services are in the pending state. – Scuba

We need to dig deep on this I guess. Would it be possible for you to share the deployment files and the logs. – Cohin

The deployment files (complete implementation) are exactly as per the example code given in the official Traefik docs. See the link at the top of the question. – Scuba

I have tried the example code using plain ingress and that works fine so the issue isn't my AWS configuration. – Scuba

Answer 3

I am trying this example and got very stupid issue as mentioned. After hours to try, I figure out that the issue is at "match" rule.

-match: PathPrefix(`/whoami-app-api`)
-match: Host(`whoami.localhost`) && PathPrefix(`/notls`)

Match without host will work by http:localhost/<prefix>. Match with a host that don't like real URI like ".net" ".com".. will work

Some host like real URI like abc.com, scx.net, the browser will try to reach it by DNS instead of local host "0.0.0.0", so it cant not reach our Traefik.

Otherwise, Host with single word won't work also. I don't know why. I think in URL, there are at least 2 words splitting by dot

Anatomist