Kubernetes Pod vs. Container OOMKilled

Question

If I understand correctly the conditions for Kubernetes to OOM kill a pod or container (from komodor.com):

If a container uses more memory than its memory limit, it is terminated with an OOMKilled status. Similarly, if overall memory usage on all containers, or all pods on the node, exceeds the defined limit, one or more pods may be terminated.

This means that if a container in the pod exceeds the total memory it will be killed (the container) but not the pod itself. Similarly, if there are multiple containers in a pod and the pod itself exceeds its memory limitation, which is the sum of memory limits of all the containers in that pod - the pod will be OOM killed. However, the latter only seems possibly if one of the containers exceeds its memory allowance. In this case - wouldn't the container be killed first?

I'm trying to understand the actual conditions in which a pod is OOM killed instead of a container.

I've also noticed that when there is one container in the pod and that container is exceeding its memory allowance repeatedly - the pod and container are killed intermittently. I observed this - the container would restart, which would be observable by watching the logs from the pod, and every second time - the pod is killed and restarted, incrementing its restart count.

If it helps to understand the behavior - the QOS class of the pod is Burstable.

Fay

How do you set the memory constraints on a pod? – Engender

@SoftwareEngineer there is only one container per pod and for that container I set resources->limits->memory: 512Mi and resources->requests->memory: 256Mi. – Fay

Answer 1

Pods aren't OOM killed at all. OOMKilled is a status ultimately caused by a kernel process (OOM Killer) that kills processes (containers are processes), which is then recognised by the kubelet which sets the status on the container. If the main container in a pod is killed then by default the pod will be restarted by the kubelet. A pod cannot be terminated, because a pod is a data structure rather than a process. Similarly, it cannot have a memory (or CPU) limit itself, rather it is limited by the sum of its component parts.

The article you reference uses imprecise language and I think this is causing some confusion. There is a better, shorter, article on medium that covers this more accurately, and a longer and much more in depth article here.

Engender

That's very insightful. So if the process going OOM is the main container in the pod - only then is the pod restarting? Could it be related to the fact that my pod is running with pid different than 1? (not using 'exec': petermalmgren.com/signal-handling-docker) – Fay

Could be, though pid 1 isn't always necessary. Even basic docker supports an --init flag that's going to force your main process away from pid 1. And, your pid in the container doesn't matter because you'll have a different pid in the kernal. – Engender

Btw, I have a pod with a sidecar that regularly fails with an OOMKilled status because it has a memory leak (which is in a 3rd party library). This suits me fine. I have other replicas running, so the only side effect is to slightly reduce my throughput while the pod automatically restarts. I achieve this by linking the pod's health to the running sidecar as well as the main container (there's another container too, acting as a network proxy, but it's never problematic). – Engender

A follow-up question: According to this response, there could be a strange condition, in which a secondary container is killed, but the pod continues to live. It seems problematic to me, as most likely the pod is not functioning as expected, and I would have expected Kubernetes to restart it. Why isn't it so? – Otes

@Otes some sidecars, init containers for example, are designed to die while the pod lives on, so the situation you describe isn't actually very strange at all. Kubernetes can't distinguish a 'mission critical' container unless you attach it to the pod health metric. You should do this for all sidecars that are tied to the pod's lifetime. – Engender

@SoftwareEngineer, but Kubernetes does have a distinction between init-containers to regular containers. Regular containers should not die, if I correctly understand. A college in my workplace actually told me that he witness the pod is restarted when any container crashes (except init-containers). Not sure if it is true or not. – Otes