What does `--oom-kill-disable` do for a Docker container?

Question

I have understood that docker run -m 256m --memory-swap 256m will limit a container so that it can use at most 256 MB of memory and no swap. If it allocates more, then a process in the container (not "the container") will be killed. For example:

$ sudo docker run -it --rm -m 256m --memory-swap 256m \
        stress --vm 1 --vm-bytes 2000M --vm-hang 0
stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [1] (415) <-- worker 7 got signal 9
stress: WARN: [1] (417) now reaping child worker processes
stress: FAIL: [1] (421) kill error: No such process
stress: FAIL: [1] (451) failed run completed in 1s

Apparently one of the workers allocates more memory than is allowed and receives a SIGKILL. Note that the parent process stays alive.

Now if the effect of -m is to invoke the OOM killer if a process allocates too much memory, then what happens when specifying -m and --oom-kill-disable? Trying it like above has the following result:

$ sudo docker run -it --rm -m 256m --memory-swap 256m --oom-kill-disable \
        stress --vm 1 --vm-bytes 2000M --vm-hang 0
stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
(waits here)

In a different shell:

$ docker stats
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
f5e4c30d75c9        0.00%               256 MiB / 256 MiB       100.00%             0 B / 508 B         0 B / 0 B           2


$ top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                           
19391 root      20   0 2055904 262352    340 D   0.0  0.1   0:00.05 stress

I see the docker stats shows a memory consumption of 256 MB, and top shows a RES of 256 MB and a VIRT of 2000 MB. But, what does that actually mean? What will happen to a process inside the container that tries to use more memory than allowed? In which sense it is constrained by -m?

Chuck

Answer 1

As i understand the docs --oom-kill-disable is not constrained by -m but actually requires it:

By default, kernel kills processes in a container if an out-of-memory (OOM) error occurs. To change this behaviour, use the --oom-kill-disable option. Only disable the OOM killer on containers where you have also set the -m/--memory option. If the -m flag is not set, this can result in the host running out of memory and require killing the host’s system processes to free memory.

A developer stated back in 2015 that

The host can run out of memory with or without the -m flag set. But it's also irrelevant as --oom-kill-disable does nothing unless -m is passed.

In regard to your update, what happens when OOM-killer is disabled and yet the memory limit is hit (intresting OOM article), id say that new calls to malloc and such will just fail as described here but it also depends on the swap configuration and the hosts available memory. If your -m limit is above the actual available memory, the host will start killing processes, one of which might be the docker daemon (which they try to avoid by changing its OOM priority).

The kernel docs (cgroup/memory.txt) say

If OOM-killer is disabled, tasks under cgroup will hang/sleep in memory cgroup's OOM-waitqueue when they request accountable memory

For the actual implementation (which docker utilizes as well) of cgroups, youd have to check the sourcecode.

Hyperactive

Sorry, I guess I should have been more explicit in my question, which seems to boil down to: What mechanism is used to limit the used memory, if it is not the OOM killer? I understand that when using -m 50m, if the container goes above that limit one process will be killed. But with -m 50m --oom-kill-disable, what will stop any process from using more than that memory limit? – Chuck

Answer 2

The job of the 'oom killer' in Linux is to sacrifice one or more processes in order to free up memory for the system when all else fails. OOM killer is only enabled if the host has memory overcommit enabled

The setting of --oom-kill-disable will set the cgroup parameter to disable the oom killer for this specific container when a condition specified by -m is met. Without the -m flag, oom killer will be irrelevant.

The -m flag doesn’t mean stop the process when it uses more than xmb of ram, it’s only that you’re ensuring that docker container doesn’t consume all host memory, which can force the kernel to kill its process. With -m flag, the container is not allowed to use more than a given amount of user or system memory.

When container hits OOM, it won’t be killed but it can hang and stay in defunct state hence processes inside the container can’t respond until you manually intervene and do a restart or kill the container. Hope this helps clear your questions.

For more details on how kernel act on OOM, check Linux OOM management and Docker memory Limitations page.

Waggoner

Answer 3

With newer docker versions on newer hosts using cgroups v2 the --oom-kill-disable argument is not supported anymore.

To check the Cgroup used by docker run

> docker info | grep Cgroup
 Cgroup Driver: systemd
 Cgroup Version: 2

When trying to use the option docker gives the following warning

WARNING: Your kernel does not support OomKillDisable. OomKillDisable discarded.

From: moby: Improve information about supported (cgroup) constraints

I am still trying to find out if there is a way arount his.

Jointless