Docker stats shows different data compared to cadvisor

Question

I have 2 Docker containers running on Centos machine

for Docker memory monitoring purpose, I have installed "cadvisor" on the machine via Docker images

when I compare "Docker stats" and "cadvisor" value then I see that cadvisor shows more memory usage than docker stats

for example, the value of memory usage in Docker stats command for one container is 520 MiB but in cadvisor value is 756 MiB

Which one is right ?

Hereinbefore

Well , both actually show the statistics on that instance , how are you determining that the stats from cAdvisor co-relates with the stats from docker for the same instance of time. – Reiko

I have integrated cadvisor with grafana and created a graph for it. On that graph, I am seeing the different value (756MiB) and on the instance when I typed "docker stats" I see the different value (520MiB) for the same container at the same time – Hereinbefore

Grafana would provide you a cumulative stats for a configured period such as 1m,5m etc , i dont think you can compare this with docker stats – Reiko

Answer 1

According to Docker docs:

On Linux, the Docker CLI reports memory usage by subtracting page cache usage from the total memory usage.

https://docs.docker.com/engine/reference/commandline/stats/

So, to get from cAdvisor the same value as docker stats show, you can use the following expression: container_memory_usage_bytes - container_memory_cache

I tried it out and got exactly the same values as docker stats produces.

Dulse

Answer 2

Well, in my case, container_memory_usage_bytes - container_memory_cache differs docker stats.

It is container_memory_working_set_bytes that matches docker stats.

$ docker stats u0a48bed
CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O        PIDS
54e1d0ab2e33   u0a48bed   0.11%     654.1MiB / 10GiB    6.39%     5.17MB / 22MB   877MB / 34.3GB   4

• container_memory_usage_bytes{name="XXX"} returns 1182535680 bytes (1127.75 MiB), matching the Total line in cAdvisor's graph
• container_memory_usage_bytes{name="XXX"} - container_memory_cache{name="XXX"} returns 122261504 bytes (116 MiB)
• container_memory_working_set_bytes{name="XXX"} returns 685834240 bytes (654.06 MiB), matching both the docker-stats value and the Hot line in cAdvisor's Graph

Gelderland