Kubernetes not getting Letsencrypt Cert

Question

I have installed microk8s, traefik and cert-manager. When I try to receive a letsencrypt certificate, a new pod for answering the challenge is created, but the request from the letsencryt server does not reach this pod. Instead, the request is forwarded to the pod that serves the website.

It looks like the ingressroute routing the traffic to the web pod has higher priority then the ingress that routes the /.well-known/acme-challenge/... requests to the correct pod. What am I missing?

kubectl edit clusterissuer letsencrypt-prod:

kind: ClusterIssuer
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cert-manager.io/v1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-prod"},"spec":{"acme":{"email":"[email protected]","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-v02.api.letsencrypt.org/directory","solvers":[{"http01":{"ingress":{"class":"traefik"}}}]}}}
  creationTimestamp: "2022-07-11T14:32:15Z"
  generation: 11
  name: letsencrypt-prod
  resourceVersion: "49979842"
  uid: 40c4e26d-9c94-4cda-aa3a-357491bdb25a
spec:
  acme:
    email: [email protected]
    preferredChain: ""
    privateKeySecretRef:
      name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress: {}
status:
  acme:
    lastRegisteredEmail: [email protected]
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/627190636
  conditions:
  - lastTransitionTime: "2022-07-11T14:32:17Z"
    message: The ACME account was registered with the ACME server
    observedGeneration: 11
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

kubectl edit ingressroute webspace1-tls:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"traefik.containo.us/v1alpha1","kind":"IngressRoute","metadata":{"annotations":{},"name":"w271a19-tls","namespace":"default"},"spec":{"entryPoints":["websecure"],"routes":[{"kind":"Rule","match":"Host(`test1.mydomain.com`)","middlewares":[{"name":"test-compress"}],"priority":10,"services":[{"name":"w271a19","port":80}]}],"tls":{"secretName":"test1.mydomain.com-tls"}}}
  creationTimestamp: "2022-10-05T20:01:38Z"
  generation: 7
  name: w271a19-tls
  namespace: default
  resourceVersion: "45151920"
  uid: 77e9b7ac-33e7-4810-9baf-579f00e2db6b
spec:
  entryPoints:
  - websecure
  routes:
  - kind: Rule
    match: Host(`test1.mydomain.com`)
    middlewares:
    - name: test-compress
    priority: 10
    services:
    - name: w271a19
      port: 80
  tls:
    secretName: test1.mydomain.com-tls

kubectl edit ingress cm-acme-http-solver-rz9mm:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2023-03-22T13:00:18Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "2306410973"
    acme.cert-manager.io/http-token: "1038683769"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-rz9mm
  namespace: default
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: test1.mydomain.com-glnrn-2096762198-4162956557
    uid: db8b5c78-8549-4f13-b43d-c6c7bba7468d
  resourceVersion: "52806119"
  uid: 6b27e02a-ee65-4809-b391-95c03f9ebb36
spec:
  ingressClassName: traefik
  rules:
  - host: test1.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-ll2zr
            port:
              number: 8089
        path: /.well-known/acme-challenge/9qtVY8FjfMIWd_wBNhP3PEPJZo4lFTw8WfWLMucRqAQ
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

get_cert.yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test1.mydomain.com
  namespace: default
spec:
  secretName: test1.mydomain.com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: test1.mydomain.com
  dnsNames:
  - test1.mydomain.com

In the webserver log of the web pod I see the reqests to /.well-known... coming in.

Pivoting

Anyone any ideas? Kubernets is unusable as hosting platform, if it does not support Lets Encrypt Certificates :-( – Pivoting

Would adding the Traefik annotation traefik.ingress.kubernetes.io/router.entrypoints: websecure help? – Genoa

@Genoa I added this annotation to the ClusterIssuer but no difference. Thanks anyway! – Pivoting

Checkout troubleshooting doc: cert-manager.io/v1.1-docs/faq/troubleshooting/…. Cert Manager will create K8S objects CertificateRequest, Order and Challenge during your Certificate validation process. kubectl describe on these objects will provide events which are quite useful to understand what happens in such situation (been there quite a few times) – Reahard

I have described the orders and challenges. I figured out that the challenge gets a 404 error. This is because the request is not routed to the right pod. – Pivoting

@Pivoting Does your clusterissuer file has a apiVersion: cert-manager.io/v1 line in top of the file? – Mom

Answer 1

Shouldn't this annotation be added to ingress?

cert-manager.io/cluster-issuer=letsencrypt-production

Claudianus

I added cert-manager.io/cluster-issuer: letsencrypt-prod to the annotations manually, but still no success. – Pivoting

Also in the entry route? Normally the secret and annotation are specified in the entry file, not in the entry route. Did you follow any guidance? – Claudianus

I used blog.zachinachshon.com/cert-manager for installing cert-manager and for the clusterissuer fosstechnix.com/kubernetes-traefik-ingress-letsencrypt #7. I added the yaml file I used to get the cert above. – Pivoting

Answer 2

Aren't you supposed to fill this "ingress {}" at the ClusterIssuer object level :

    solvers:
    - http01:
        ingress: {}

According to this doc, they are adding a class, maybe that mean you will help your resolver-pod to announce correctly to your ingress-controller. I don't have a cluster next to me, sorry i cannot test this solution, i hope it helps :)

Strangulation

Yes, I also tried with ingress: Class: traefik – Pivoting