Get Docker Container CPU Usage as Percentage

Question

Docker provides an interactive stats command, docker stats [cid] which gives up to date information on the CPU usage, like so:

CONTAINER      CPU %          MEM USAGE/LIMIT       MEM %       NET I/O
36e8a65d       0.03%          4.086 MiB/7.798 GiB   0.05%       281.3 MiB/288.3 MiB

I'm trying to get the CPU usage as a percentage in a digestible format to do some analysis.

I've seen the stats in /sys/fs which seem to provide similar values as the Docker Remote API which gives me this JSON blob:

{
    "cpu_usage": {
        "usage_in_usermode": 345230000000, 
        "total_usage": 430576697133, 
        "percpu_usage": [
            112999686856, 
            106377031910, 
            113291361597, 
            97908616770
        ], 
        "usage_in_kernelmode": 80670000000
    }, 
    "system_cpu_usage": 440576670000000, 
    "throttling_data": {
        "throttled_time": 0, 
        "periods": 0, 
        "throttled_periods": 0
    }
}

But I'm unsure how to get an exact CPU Usage as a percentage from that.

Any ideas?

Scrape

Isn't your value in the example above as percentage or am I getting something wrong? – Neusatz

He meant that how he could get percentage from the api, in the same manner as how its displayed by the docker client – Annecy

Answer 1

If you are going to use the Stats API call - you can take a look at how the docker client does it: https://github.com/docker/docker/blob/eb131c5383db8cac633919f82abad86c99bffbe5/cli/command/container/stats_helpers.go#L175-L188

func calculateCPUPercent(previousCPU, previousSystem uint64, v *types.StatsJSON) float64 {
    var (
        cpuPercent = 0.0
        // calculate the change for the cpu usage of the container in between readings
        cpuDelta = float64(v.CPUStats.CPUUsage.TotalUsage) - float64(previousCPU)
        // calculate the change for the entire system between readings
        systemDelta = float64(v.CPUStats.SystemUsage) - float64(previousSystem)
    )

    if systemDelta > 0.0 && cpuDelta > 0.0 {
        cpuPercent = (cpuDelta / systemDelta) * float64(len(v.CPUStats.CPUUsage.PercpuUsage)) * 100.0
    }
    return cpuPercent
}

Basically, you take a point of reference, then see the difference in say 10 secs, you can then tell how much of the time was used by the container. Say, we start with 0 SystemCPUUsage and 0 CPUUsage for the container. If after 10 secs, we have 10 SystemCPUUsage and 1 CPUUsage, then we have 10% usage. You are just given the results in nanoseconds, not seconds, in the API. The actual time does not matter, the total SystemCPUUsage change is what matters, then compare CPUUSage to that.

Osterman

above link is broken. found it at: github.com/docker/docker/blob/… – Silicosis

Worth noting is that v.CPUStats.CPUUsage.PercpuUsage is an array containing the usage of all your CPU-cores. Unfortunately it returns a value of 0 if the cores are not in use. This means that percentages will be skewed if docker is listing cores that aren't used. I found this out the hard way when my percentages weren't adding up. Also, cpu_stats.online_cpus is available in newer versions of the API so you can just use that instead. – Taam

Answer 2

After we consume the remote api we get these fields: precpu_stats/cpu_stats

Then, basically here is the code: (javascript example)

var res <---- remote api response
    
var cpuDelta = res.cpu_stats.cpu_usage.total_usage -  res.precpu_stats.cpu_usage.total_usage;
var systemDelta = res.cpu_stats.system_cpu_usage - res.precpu_stats.system_cpu_usage;
var RESULT_CPU_USAGE = cpuDelta / systemDelta * 100;

Just to clarify the RESULT_CPU_USAGE... it's the amount of resource consumed from your physical hardware, so supposing you are getting RESULT_CPU_USAGE as 50%, it means that 50% of all your PC power is being used by container X

Thirtyeight

Your calculation is not taking number of CPUs into account. – Ciel

I think it doesn't make much sense to multiply the result by the numbers of CPU. I mean... my notebook has 4 cores, if I stress these cores from a docker container and then run docker stats from outside... I get ~400% of cpu usage. If I am monitoring a docker host, I'd rather have a range from 0% to 100% – Thirtyeight

This cpu usage percentage appears to be a second. Is there a way to get the average cpu used by the container for a minute without hitting this api call 60 times a minute? – Churr

did anyone get the right calculation? This answer is not correct. – Empressement

@Empressement answer given by Michael gives the correct calculation. I have verified that the result is similar to docker stats CLI's output – Woolly

Answer 3

So I need this also, and the following gives me the correct CPU usage, factoring in number of cores.

var cpuDelta = metric.cpu_stats.cpu_usage.total_usage -  metric.precpu_stats.cpu_usage.total_usage;
var systemDelta = metric.cpu_stats.system_cpu_usage - metric.precpu_stats.system_cpu_usage;
var RESULT_CPU_USAGE = cpuDelta / systemDelta * metric.cpu_stats.cpu_usage.percpu_usage.length * 100;

console.log(RESULT_CPU_USAGE);

Kirima

Answer 4

UPDATE

This formula changes depending on your docker verison For example, verison 24.0.9, the "percpu_usage" variable will be nil

Please refer to the version in the docker/cli repo

Updated:

func calculateCPUPercentUnix(previousCPU, previousSystem uint64, v *types.StatsJSON) float64 {
    var (
        cpuPercent = 0.0
        // calculate the change for the cpu usage of the container in between readings
        cpuDelta = float64(v.CPUStats.CPUUsage.TotalUsage) - float64(previousCPU)
        // calculate the change for the entire system between readings
        systemDelta = float64(v.CPUStats.SystemUsage) - float64(previousSystem)
        onlineCPUs  = float64(v.CPUStats.OnlineCPUs)
    )

    if onlineCPUs == 0.0 {
        onlineCPUs = float64(len(v.CPUStats.CPUUsage.PercpuUsage))
    }
    if systemDelta > 0.0 && cpuDelta > 0.0 {
        cpuPercent = (cpuDelta / systemDelta) * onlineCPUs * 100.0
    }
    return cpuPercent
}

https://github.com/docker/cli/blob/925e7d687045ea0c5a674c830762b7787c3253b1/cli/command/container/stats_helpers.go#L166

PercpuUsage being 0 makes the cpu percentage 0

Billiot