Finding which process was killed by Linux OOM killer [closed]

Question

When Linux runs out of memory (OOM), the OOM killer chooses a process to kill based on some heuristics (it's an interesting read: http://lwn.net/Articles/317814/).

How can one programmatically determine which processes have recently been killed by the OOM killer?

Sorcerer

Answer 1

Try this out:

grep -i 'killed process' /var/log/messages

Sedgewick

FWIW, I get those messages in syslog, or kern.log, but not /var/log/messages – Boiler

You can use "egrep -i -r 'killed process' /var/log/" to search it also in other places. – Scrawly

@jberryman: For some reason, syslog is in /var/log/syslog on some distros, and /var/log/messages on others. I think it's Debian for the former and Red Hat for the latter, BICBW. – Fibrinolysis

" dmesg | egrep -i 'killed process' " and you can search logs anywhere (including archived ones) :) – Benenson

egrep doesn't make sense here. Plain old grep, or if we're being specific, fgrep, makes much more sense. (Editing answer accordingly.) – Paoting

The precise location depends on the logging arcitecture and its configuration. /etc/syslog.conf is a good place to start looking, though on some arcane systems, this too could be in a different location. – Wherewith

Answer 2

Try this so you don't need to worry about where your logs are:

dmesg -T | egrep -i 'killed process'

-T, --ctime - Print human-readable timestamps.

Delly

This is also useful, but while I unfortunately can't explain it, I'm seeing results in /var/log/messages that aren't showing up in dmesg//var/log/dmesg. It could be some sort of misconfiguration, but worth noting that using both approaches could be a good idea. – Botryoidal

Not sure about your log file, but the output of dmesg is from a limited-size ring buffer. If other things have filled the buffer since the oom-killer then you'll lose the oom-killer output. – Medallist

This was the only way I found how to see that process was killed in OpenVZ container – Ibnsaud

Compared to /var/log/messages, this has the advantages of not requiring root privileges – Robyn

For me it was just at the bottom of dmesg -T. I didn't need to grep. – Ruthenic

Please keep in mind dmesg is for kernel ring buffer, not the messages before start or reboot – Herewith

Answer 3

Now dstat provides the feature to find out in your running system which process is candidate for getting killed by oom mechanism

dstat --top-oom
 --out-of-memory---
  kill score
 java           77
 java           77
 java           77

and as per man page

  --top-oom
          show process that will be killed by OOM the first

Scrawl

This info is meaningless without knowing what the score means, and that's not documented anywhere. All you might see is the score increase, then the process being killed, so maybe it was the oom killer, or maybe it was something else, there's no way to be sure. – Catechize

Where does dstat read this kill score data from? (A file?) Edit: Just found: /proc/<PID>/oom_score – Tinaret

Answer 4

Try this out:

grep "Killed process" /var/log/syslog

Paz