what does "MEM USAGE / LIMIT " column state in docker stats command output?

Question

I created a docker container from an image which is of size: ~90 MB.

The container is running fine. I wanted to know how much RAM does a container must be using on its host machine. So I ran "docker stats" command and it shows following output:

CONTAINER ID        NAME                   CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
66c266363746        sbrconfigurator_v0_2   0.00%               32.29MiB / 740.2MiB   4.36%               0B / 0B             15.1MB / 4.1kB      10

Here it shows memory usage as followings:

MEM USAGE / LIMIT
32.29MiB / 740.2MiB

I don't know what does this 740.2 MB memory means, whether it means that 740.2 MB of RAM has been allocated to this container i.e 740.2 MB RAM is being used by this container or not.

Please help me know how much RAM (host's machine) does this container must be using. Host machine is Linux, Ubuntu.

Monocoque

Have a look at Docker reserve a certain amount of memory for container, this might implicitly answer your question – Propylite

Answer 1

The memory limit shows how much memory docker will allow the container to use before killing the container with an OOM. If you do not set the memory limit on a container, it defaults to all of the memory available on the docker host. With Docker for Win/Mac, this is memory allocated to the embedded VM which you can adjust in the settings. You can set the memory limit in the compose file, or directly on the docker run cli:

$ docker run -itd --name test-mem -m 1g busybox /bin/sh
f2f9f041a76c0b74e4c6ae51dd57070258a06c1f3ee884d07fef5a103f0925d4

$ docker stats --no-stream test-mem
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O BLOCK I/O           PIDS
f2f9f041a76c        test-mem            0.00%               360KiB / 1GiB       0.03%               5.39kB / 0B 3.29MB / 0B         1

In the above example, busybox is not using 1GB of memory, only 360KB.

Without setting the limit, the memory limit in GiB can be converted (GiB*1024 = KB) to show something very close to what you see in the free command for total memory on the host. Not sure if the small difference between the two accounts for the kernel or some other overhead.

Ferris

I am trying to figure out where docker stats is pulling the MEM USAGE value from... Can you help? – Spillar

with Docker desktop, under the configuration, Resources, one can change the memory settings. – Fates

@PaulMichalik the usage value should be extracted from the cgroup of the container, just as cgroups are used to set the limits. – Ferris

Should be visible inside the container at cat /sys/fs/cgroup/memory/memory.usage_in_bytes but I'm not at a machine to test right now. – Ferris

@PaulMichalik Given the example above the file should be located in /sys/fs/cgroup/memory/docker/f2f9f041a76c0b74e4c6ae51dd57070258a06c1f3ee884d07fef5a103f0925d4/memory.stat (sum cache + rss + swap). Value in memory.usage_in_bytes isn't necessarily precise. – Croce

Answer 2

Image size of 90MB is only size on disk, it doesn't have much to do with occupied memory (RAM).

When you start a Docker container on Linux, e.g.:

$ docker run -itd --name test busybox /bin/sh
74a1cb1ecf7ad902c5ddb842d055b4c2e4a11b3b84efd1e600eb47aefb563cb3

Docker will create a directory in cgroups fs, typically mounted in /sys/fs/cgroup under the long ID of the container.

/sys/fs/cgroup/memory/docker/74a1cb1ecf7ad902c5ddb842d055b4c2e4a11b3b84efd1e600eb47aefb563cb3/

In order to make the path shorter I'll write

/sys/fs/cgroup/memory/docker/$(docker inspect -f "{{.Id}}" test)  

instead. The directory contains number of plain text files read/written by the kernel.

When you run Docker container without -m / --memory flag, the container would be able to allocate all memory available on your system, which would be equal to:

$ numfmt --to=iec $(cat /sys/fs/cgroup/memory/docker/memory.max_usage_in_bytes)

When you specify a memory limit e.g. -m 1g

$ docker run -itd --name test -m 1g busybox /bin/sh

Docker will write the limit into file memory.limit_in_bytes:

$ numfmt --to=iec $(cat /sys/fs/cgroup/memory/docker/$(docker inspect -f "{{.Id}}" test)/memory.limit_in_bytes)
1.0G

This tells the Linux kernel to enforce hard memory limit on the container. In case that the container is trying to allocate more memory than the limit, the kernel will invoke the infamous OOM killer.

The "MEM USAGE" is probably read from memory.usage_in_bytes, which is approximation of actual memory usage.

$ numfmt --to=iec $(cat /sys/fs/cgroup/memory/docker/$(docker inspect -f "{{.Id}}" test)/memory.usage_in_bytes)
312K

According to cgroups documentation more precise value can be obtained from memory.stat file.

If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).

Where you'd have to sum those 3 lines. Normally memory.usage_in_bytes is good enough estimate.

Croce

Answer 3

For those looking to calculate the docker stats memory themselves, per this docker doc https://docs.docker.com/engine/reference/commandline/stats/

On Linux, the Docker CLI reports memory usage by subtracting cache usage from the total memory usage. The API does not perform such a calculation but rather provides the total memory usage and the amount from the cache so that clients can use the data as needed. The cache usage is defined as the value of total_inactive_file field in the memory.stat file on cgroup v1 hosts. On Docker 19.03 and older, the cache usage was defined as the value of cache field. On cgroup v2 hosts, the cache usage is defined as the value of inactive_file field.

The the total memory usage referenced above appears to come from memory.usage_in_bytes file located in the same folder as memory.stat

Mercurate