How to get number of pods running in prometheus

Question

I am scraping the kubernetes metrics from prometheus and would need to extract the number of running pods.

I can see container_last_seen metrics but how should i get no of pods running. Can someone help on this?

Advance

I'm sorry but I do not understand what do You mean. What does "no" mean in this sentence "How to get no of pods running in prometheus"? – Hesketh

I would assume that you need to scrape the number of pods in each namespace and then add all these numbers together? – Langill

Yes i need to export number of pods running . – Advance

Answer 1

If you need to get number of running pods, you can use a metric from the list of pods metrics https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md for that (To get the info purely on pods, it'd make sens to use pod-specific metrics). For example if you need to get the number of pods per namespace, it'll be: count(kube_pod_info{namespace="$namespace_name"}) by (namespace) To get the number of all pods running on the cluster, then just do: count(kube_pod_info)

Aguila

Answer 2

Assuming you want to display that in Grafana according to your question tags, from this Kubernetes App Metrics dashboard for example:

count(count(container_memory_usage_bytes{container_name="$container", namespace="$namespace"}) by (pod_name))

You can just import the dashboard and play with the queries.

Depending on your configuration/deployment, you can adjust the variables container_name and namespace, grouping by (pod_name) and count'ing it does the trick. Some other label than pod_name can be used as long as it's shared between the pods you want to count.

Popovich

Thanks , i saw this but it is lil bit confusing about the github.com/prometheus/prometheus/blob/master/documentation/… . My kubernetes and prometheus are on different server. – Advance

Answer 3

As of the latest updates in Prometheus monitoring for Kubernetes, the metric you might be looking for to count the number of running pods on each node is kubelet_running_pods. This metric is exposed by the Kubelet’s Prometheus endpoint and gives a real-time count of pods that are actively running on each node.

This metric is more useful for operational purposes like monitoring and alerting on pod capacity and usage in real-time. To fetch the number of running pods using the kubelet_running_pods metric, you can use a simple Prometheus query like: sum(kubelet_running_pods) by (node)

Drye

Answer 4

If you want to see only the number of "deployed" pods in some namespace, you can use the solutions in previous answers.

My use case was to see the current running pods in some namespace and below is my solution:

'min_over_time(sum(group(kube_pod_container_status_ready{namespace="BC_NAME"}) by (pod,uid)) [5m:1m]) OR on() vector(0)'

Please replace BC_NAME with your namespace name. The timespan provides you fine the data.

If no data found - no pod currently running it returns '0'

Turkmen

This doesn't work for me. I have one pod in the kube_pod_container_status_ready{} result, which I re-deployed, then the result of this query is 2, not 1, because two pods have been ready at any point within the query. I would expect this query to give the number of current running pods (1), not 2. – Gymnastics