prometheus operator - enable monitoring for everything in all namespaces

Question

I want to monitor a couple applications running on a Kubernetes cluster in namespaces named development and production through prometheus-operator.

Installation command used (as per Github) is:

helm install prometheus-operator stable/prometheus-operator -n production --set prometheusOperator.enabled=true,prometheus.service.type=NodePort,prometheusOperator.service.type=NodePort,alertmanager.service.type=NodePort,grafana.service.type=NodePort,grafana.service.nodePort=30906

What parameters do I need to add to above command to have prometheus-operator discover and monitor all apps/services/pods running in all namespaces?

With this, Service Discovery only shows some prometheus-operator related services, but not the app that I am running within 'production' namespace even though prometheus-operator is installed in the same namespace.

Anything I am missing?

Note - Am running performing all actions using the same user (which uses the $HOME/.kube/config file), so I assume permissions are not an issue.

kubectl version - v1.17.3 helm version - 3.1.2

P.S. There are numerous articles on this on different forums, but am still not finding simple and direct answers for this.

Homogenize

What is your env: local (Minikube, Kubeadm), On-Prem? What metrics you want to get? CPU/Memory metrics or more complex? Do you have custom.metrics in your cluster? – Jessjessa

On-prem kubeadm it is (not minikube). I would like to see basic CPU, Memory related metrics on Prometheus and Grafana before thinking abt complex ones. At this point, nothing is shown related to the app's services or pods in Prometheus UI. – Homogenize

How many nodes you have in kubeadm cluster? Also did your metrics-server gahter information? Few days ago I had issue with service discovery which was related with metrics server (if you have 2 or more nodes you need to modify metrics-server deployment). Could you check this and provide your metrics-server YAML? Also logs from metrics server with --v=6 – Jessjessa

Answer 1

I had the same problem. After some investigation answering with more details.

I've installed Prometheus stack via Helm charts which include Prometheus operator chart directly as a sub-project. Prometheus operator monitors namespaces specified by the following helm values:

prometheusOperator:
  namespaces: ''
  denyNamespaces: ''
  prometheusInstanceNamespaces: ''
  alertmanagerInstanceNamespaces: ''
  thanosRulerInstanceNamespaces: ''

The namespaces value specifies monitored namespaces for ServiceMonitor and PodMonitor CRDs. Other CRDs have their own settings, which if not set, default to namespaces. Helm values are passed as command-line arguments to the operator. See here and here.

Prometheus CRDs are picked up by the operator from the mentioned namespaces, by default - everywhere. However, as the operator is designed with multiple simultaneous Prometheus releases in mind, what to pick up by a particular Prometheus app instance is controlled by the corresponding Prometheus CRD. CRDs selectors and corresponding namespaces selectors are controlled via the following Helm values:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: true
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

Similar values are present for other CRDs: alertmanagerConfigXXX, ruleNamespaceXXX, podMonitorXXX, probeXXX. XXXSelectorNilUsesHelmValues set to true, means to look for CRD with particular release label, e.g. release=myrelease. See here.

Empty selector (for a namespace, CRD, or any other object) means no filtering. So for Prometheus object to pick up a ServiceMonitor from the other namespaces there are few options:

• Set serviceMonitorSelectorNilUsesHelmValues: false. This leaves serviceMonitorSelector empty.
• Apply the release label, e.g. release=myrelease, to your ServiceMonitor CRD.
• Set a non-empty serviceMonitorSelector that matches your ServiceMonitor.

For the curious ones here are links to the operator sources:

• Enqueue of Prometheus CRD processing
• Processing of Prometheus CRD

Ahab

Answer 2

I used values.yaml from https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml, modified parameters *NilUsesHelmValues to False and it seems to work fine with that. helm install prometheus-operator stable/prometheus-operator -n monitoring -f values.yaml

Also, like https://stackoverflow.com/users/7889479/anish-kumar-mourya stated, the services do show in Grafana dashboard even though they dont appear in Prometheus UI under Service Discovery or Targets.

Hope this helps other newbies like me.

Homogenize

Answer 3

You need to create a service for the pod and a serviceMonitor custom resource to configure which services in which namespace need to be discovered by prometheus.

kube-state-metrics Service example

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kube-state-metrics
    k8s-app: kube-state-metrics
  annotations:
    alpha.monitoring.coreos.com/non-namespaced: "true"
  name: kube-state-metrics
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: metrics
    protocol: TCP
  selector:
    app: kube-state-metrics

This Service targets all Pods with the label k8s-app: kube-state-metrics.

Generic ServiceMonitor example

This ServiceMonitor targets all Services with the label k8s-app (spec.selector) any value, in the namespaces kube-system and monitoring (spec.namespaceSelector).

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: k8s-apps-http
  labels:
    k8s-apps: http
spec:
  jobLabel: k8s-app
  selector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
  namespaceSelector:
    matchNames:
    - kube-system
    - monitoring
  endpoints:
  - port: http-metrics
    interval: 15s

https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md

Paunchy

just not for someone that need to find how to match all namespace namespaceSelector{any:true} – Stonedeaf

Answer 4

no its fine but you can create new namespace for monitoring and install prometheus over there would be good to manage things related to monitoring.

helm install prometheus-operator stable/prometheus-operator -n monitoring

Ainsworth

But why are the apps under Production and Development namespaces not getting discovered and shown under Service Discovery or Targets? – Homogenize

its discovering bud, go to Home , select "kubernetes-compute-resources-pod" existing dashboard , and see namespace options there. – Ainsworth

In My course, I talked about the integration between prometheus-operator and others ( nginx-ingress, cluster-autoscaler,... ) – Knut

check prometheusOperator.namespaces in helm chart option, by default it should take all namespaces . whats status of helm installation on your cluster ? because its working my end. – Ainsworth