Is there any way to represent POD CPU usage in terms of CPU cores using prometheus metrics

Question

I just want to represent the CPU metrics of a POD as below

I am able to represent the CPU requests and limits in terms of CPU cores which are directly available through prometheus scrape metrics.

But in prometheus I don't see any direct metric to get the CPU cores used by a POD so, Can someone give me a work around or a way to represent the CPU usage of a POD in the terms of CPU cores.

Thanks in advance

Mimosaceous

Answer 1

The query you are looking for is this one:

sum(rate(container_cpu_usage_seconds_total{container_name!="POD"}[1m])) by (pod_name)

Here the explanation (from within to out, as Prometheus calculates this query):

• container_cpu_usage_seconds_total which gives you how long the CPU has been occupied. 1s = a single Core for a whole second
• {container_name!="POD"} ignores the meta cGroups.
• rate(....[1m]) gives you the value changes in the defined interval, here 1 minute
• sum(....) by (pod_name) add up all values which contain the same pod names, which will be the case if we have multiple containers in the same pod

For further information about Prometheus in Kubernetes you can read this blog here: https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e66

Bismuthinite

thanks for the info but when I apply above principle I observed the following things as shown in the below graph [pod cpu uage]: i.sstatic.net/vV16Q.jpg though the cpu usage is not at near the cpu limit (even after I multiply the resultant value by 100) the cpu started throttling. So I think the above principle you have mentioned is not accurate or please help me understand If I am doing anything wrong – Mimosaceous

by the way below is my pod resource config resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" – Mimosaceous

the above Prometheus query works fine, but somehow in Minikube, the pod will start throttling even though the CPU usage is not near the defined limits. – Mimosaceous

Answer 2

The container_name and pod_name labels have been renamed to container and pod starting from Kuberntes 1.16 - see this issue for details. So for Kubernetes 1.16+ the following PromQL query must be used for calculating per-pod CPU usage in CPU cores:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod)

The filter on countainer!="" is needed because cadvisor exposes container_cpu_usage_seconds_total metric for the hierarchy of containers without the container label. The query will return inflated results without the container!="" filter. See this answer for more details.

Sheeree