Is there a way to monitor tls certificates in kubernetes using prometheus?

Question

I want to monitor my tls certificates in Kubernetes using Prometheus and get a dashboard in grafana. I want to monitor their expiry and would want to get an alert when the certificates are going to be expired in 30 days. I did a lot of research and I finally found https://github.com/enix/x509-exporter. How do I use it? Is there any other efficient way to monitor the expiry of the certificates?

Afrika

Answer 1

DISCLAIMER: I haven't tried this x509-exporter. Just giving suggestion as per my understanding.

The README file seems bit off. The first thing you need to do is create a github issue, no worries I raised one here.

I am listing down steps as per my understanding and referring the usage section.

• Use their official docker image and deploy it as a deployment on k8s.
• Check sample k8s yaml files for creating deployment. Also note that the deployment yaml should mount a host directory where all the k8s certificates are stored.
• As per documentation, usually the certificates are located at /etc/kubernetes/pki.
• The deployment yaml should contain a command where you point the exporter to the directory where certificates are located with other necessary options. Like this

command: ["x509-exporter"]
args: ["-d", "/etc/kubernetes/pki", "-p", "8091", "--debug"]

Note: Here I am running exporter in debug mode on port 8091, remember to expose this port.

• In prometheus config, add the x509-exporter endpoint as target to scrape the metrics and plot those by creating graphs in Grafana dashboard.

Wrapped

That was a really detailed answer. Yes using this I can monitor the certs located at a folder. Let us assume that I don't have the certs in my machine and I want to monitor multiple tls secrets. How do I do that? Is there a way to do that? And FYI I am using prometheus operator – Afrika

FOr this exporter specifically you got to have the certs to monitor it. If certs are spread across different machines then either run multiple exporters or consolidate all certs in specific single location and then monitor it. – Wrapped

I have tried making a grafana dashboard, it gives me a lot of unnecessary info. The info that I would be needing is no_of_days it is gonna expire in, the name of the file, and the CN of the certificate. I am new to writing queries. Can you help me out? – Afrika

You need to create your own grafana dashboards with the queries referring to the required metrics like no_of_days, etc. For starters check this prometheus.io/docs/visualization/grafana/… – Wrapped

I have created deployment using exporter image, (pv, pvc - hostpath -etc/kubernetes/pki ), sevice to expose container port 8091. The pod error container creating state. When I checked the events it says executable file not found in the path. Where am I missing? – Mckenna

Answer 2

Another way is to install the x509-exporter using the helm chart : https://hub.helm.sh/charts/enix/x509-exporter

See documentation here https://github.com/enix/helm-charts/tree/master/charts/x509-exporter.

You might also find the following prometheus alert rules useful (based on the x509-exporter metrics):

check-kubernetes-certificate.rules.yml :

groups:
- name: check-kubernetes-certificate-expiration.rules
  rules:
  - alert: KubernetesCertificateExpiration
    expr: floor((x509_cert_not_after - time()) / 86400) < 90
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
      description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'

  - alert: KubernetesCertificateExpirationCritical
    expr: floor((x509_cert_not_after - time()) / 86400) < 10
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
      description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'

  - alert: KubeletCertificateEmbedded
    expr: x509_cert_not_after{filename="kubelet.conf", embedded_kind="user"}
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: '{{ $labels.instance }}: Embedded certificate in {{ $labels.filename }}'
      description: '{{ $labels.nb_cluster }} has kubelet {{ $labels.subject_CN }} running with an embedded certificate in {{ $labels.filepath }}'

Linders

Thank you!! Yes, this seems to be fine but it does not have a grafana dashboard. I have found another one with granafa dashboard. github.com/joe-elliott/cert-exporter/tree/master/docs/examples/… – Afrika

The author’s Grafana dashboard is available from the official Grafana Dashboard repository. – Tenter

Answer 3

The official prometheus/blackbox_exporter have the ssl cert expiry info already.

 Name: "probe_ssl_earliest_cert_expiry",
 Help: "Returns earliest SSL cert expiry date",

So all you need is:

1. Setup blackbox_exporter and the Probe rules to the domain you want to monitor.
   You can check my project kehao95/helm-prometheus-exporter to install blackbox_exporter via helm chart.
2. config rule to monitor certificate expiring.

You can config your prometheusRule like this: (assuming you're using prometheus-operator)

rules: 
  - alert: TLS certificate expiring
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 45
    labels:
      severity: warning
  - alert: TLS certificate expiring
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 30
    labels:
      severity: critical

Initial