Prometheus queries to get CPU and Memory usage in kubernetes pods

Question

I need to get CPU and Memory usage in kubernetes pods with prometheus queries. Can someone plz help?

Ribose

Please provide more information on your current situation. Is Prometheus up and running but you don't know how to query for metrics? Are you having trouble getting Prometheus running in your cluster? – Untutored

Possible duplicate of count k8s cluster cpu/memory usage with prometheus – Bifoliolate

yes. it's up and running I want get alerts for CPU and Memory usage of the pods. For that I need to have prometheus queries. Plz can I have what u r using ? – Ribose

I want to have something like this "sum(container_memory_usage_bytes{namespace="$namespace", pod_name="$pod", container_name!="POD"}) by (container_name)" Since there are variables in this query Im unable to send alerts. – Ribose

Please edit your question with whatever query you tried. – Linette

Answer 1

For CPU percentage

avg((sum (rate (container_cpu_usage_seconds_total {container_name!="" ,pod="<Pod name>" } [5m])) by (namespace , pod, container ) / on (container , pod , namespace) ((kube_pod_container_resource_limits_cpu_cores >0)*300))*100)

For Memory percentage

avg((avg (container_memory_working_set_bytes{pod="<pod name>"}) by (container_name , pod ))/ on (container_name , pod)(avg (container_spec_memory_limit_bytes>0 ) by (container_name, pod))*100)

you can use above promql with pod name in a query.

Dronski

why are you multiplying with 300 – Vociferate

Answer 2

The following query should return per-pod number of used CPU cores:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) without (container)

The following query should return per-pod RSS memory usage:

sum(container_memory_working_set_bytes{container!=""}) without (container)

See this answer about container!="" filter in queries above.

If you need summary CPU and memory usage across all the pods in Kubernetes cluster, then just remove without (container) suffix from queries above.

Vehicle

Answer 3

To better monitor the memory usage of Pod/Containers. The container_memory_max_usage_bytes should also be used to monitor besides container_memory_working_set_bytes.

---

We used to monitor the memory usage of the pod through

avg(container_memory_working_set_bytes{container!="POD"}) by (pod)  / (avg(kube_pod_container_resource_requests_memory_bytes{container!="POD"} > 0) by (pod)) *100 

However, we met the OOMKilled of the pod and failed to find anything abnormal from the above metrics.

---

Through the container_memory_max_usage_bytes we find the abnormal memory usage of the pod.

avg(container_memory_max_usage_bytes{ container!="POD"}) by (pod)  / (avg(kube_pod_container_resource_requests_memory_bytes{ container!="POD"} > 0) by (pod)) *100 

Ardehs

Answer 4

Do you use prometheus-operator to collect data from kubernetes? If yes, you can use something like this: sum(container_memory_usage_bytes) sum(container_cpu_usage_seconds_total) Just for example.

Cyclamate

I want to have something like this "sum(container_memory_usage_bytes{namespace="$namespace", pod_name="$pod", container_name!="POD"}) by (container_name)" Since there are variables in this query Im unable to send alerts. – Ribose