Something keeps killing my Java process on Ubuntu, anyone know why?

Question

So every couple of days my java process on Ubuntu is killed automatically, and I can't figure out why.

My box has 35.84 GB of RAM, when I launch my Java process I pass it the -Xmx28g parameter, so it should be using way less than the maximum RAM available.

I ran jstat as follows:

# jstat -gccause -t `pgrep java` 60000

The last few lines of output from jstat immediately before the process was killed were:

Time     S0     S1     E      O      P       YGC   YGCT       FGC FGCT     GCT     LGCC                 GCC
14236.1  99.98   0.00  69.80  99.40  49.88   1011  232.305    11  171.041  403.347 unknown GCCause      No GC
14296.2  93.02   0.00  65.79  99.43  49.88   1015  233.000    11  171.041  404.041 unknown GCCause      No GC
14356.1  79.20   0.00  80.50  99.55  49.88   1019  233.945    11  171.041  404.986 unknown GCCause      No GC
14416.2   0.00  99.98  24.32  99.64  49.88   1024  234.945    11  171.041  405.987 unknown GCCause      No GC

This seems to be what went down in the /var/log/syslog around this time: https://gist.github.com/1369135

There is really nothing running on this server other than my java app. What's going on?

edit: I'm running java version 1.6.0_20, the only notable parameters I'm passing to java on startup are "-server -Xmx28g". I'm not using an application server but my app embeds the "Simple web framework".

Oudh

Max physical ram doesn't equate to how much a process can utilise. Eric Lippert had a great post on this <blogs.msdn.com/b/ericlippert/archive/2009/06/08/…>. I know the post is Windows/.NET centric, but it is elightening also. Just out of curiosity, can you attempt to catch an OutOfMemoryError and log this to confirm/deny that this is the cause? – Kabuki

I'm logging stdout and stderr, which I believe is where an OOM would go, and I don't see anything that would indicate an OOM exception... In my experience an OOM results in the app quitting, not being killed. In this case it appears that the app was killed by the OS. – Oudh

Answer 1

Assuming the problem is the OOM killer, then it has killed your process in a desperate attempt to keep the OS functioning in a severe memory shortage crisis.

I would conclude that:

• your JVM is actually using significantly more than 28Gb; i.e. you've got significant non-heap memory usage, and

• the OS is not configured with an adequate amount of swap space.

I'd try adding more swap space, so that the OS can swap out parts of your application in an emergency.

Alternatively, reduce the JVM's heap size.

---

Note that "-Xmx ..." sets the maximum heap size, not the maximum amount of memory that your JVM can use. The JVM puts some stuff outside the heap, including such things as the memory for thread stacks and memory-mapped files that your application is using.

---

The syslog confirms that it is the OOM killer at work.

In what way does the linked syslog say so?

It says this:

Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606133] Out of memory: kill process 6368 (run.sh) score 4747288 or a child
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606146] Killed process 9359 (java)

The console says that java was killed, not that it quit.

Correct. It was killed by the operating system's OOM killer.

If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't.

That is what would have happened if you had filled up the Java heap.

That is not what is going on here. The actual problem is that there is not enough physical RAM to hold the Java heap. The OOM killer deals with it ...

I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM.

Unfortunately, you are trying to use way more RAM than is available on the system. This is causing virtual memory to thrash, affecting the entire operating system.

When the system starts to thrash badly, the OOM killer (not the JVM) identifies your Java process as the cause of the problem. It then kills it (with a SIGKILL) to protect the rest of the system. If it didn't, there is a risk that the entire system would lock up completely and need to be hard rebooted.

---

Finally, you said:

My box has 35.84 GB of RAM ...

That is rather a strange value. 32 GiB is 34,359,738,368 bytes or 34.35 GB.

But based on that and the observed behavior, I suspect that that is the available virtual memory rather than physical RAM. Alternatively, your "box" could be a virtual machine with RAM overcommit enabled at the hypervisor level.

Tephra

In what way does the linked syslog say so? The console says that java was killed, not that it quit. If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't. I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM. – Oudh

Per some other advice I'm reducing the max memory usage to 15GB – Oudh

Answer 2

Welcome to the OOM-killer, a linux 'feature' that is the bane of large-memory applications everywhere. There's no simple recipe to deal, just google for it and start reading and weaping.

While I can't put my mental fingers on a concise explanation of the shenigans of the OOM killer, I recall that the critical tuning parameter is called 'swappiness'. On one of our big servers, we have:

/etc/sysctl.conf:vm.swappiness=20

Read http://www.gentooexperimental.org/~patrick/weblog/archives/2009-11.html.

Shipmate

Can you summarize why the OOM-killer would kill an app using only 28GB on a machine with 35GB of RAM and basically no other processes running on it? – Oudh

see also #15237567 – Shovel

The link seems to be broken – Ensheathe

Answer 3

What JVM are you using? and what application server? It's possible that you're allocating way too much memory, and that can be problematic - the garbage collector might have trouble doing its job.

I'm not sure if this is your case, but I found quite interesting this article explaining the way Linux overcommits memory.

Meperidine

Updated my question with the info you requested – Oudh

Answer 4

wow, can you actually have 28 GB of heap?! May be you should try reducing it, keep it at no more than 50% of the RAM I think (so ~18 GB, or may be even 15 GB). Plus 171 Full GCs are a lot! How long was this app running? 171 in 2-3 days sounds huge. btw the gist indicates an OOM before termination - I think reducing the heap will fix it ( you may be limiting the JVM from expanding native space). Try adjusting various parameters, try stack size for example (-Xss) if needed. Check max perm size and other sections too. Its a memory problem and it may not necessarily be the heap.

Pederson

oops misread, its actually 11 full GCs, but thats still quite a few. its definitely a OOM and you should try reducing the heap size. – Pederson

Sorry, I'm not seeing where the gist indicates an OOM, I can't see any OOM in stdout or stderr for the process :-/ Why is it important that the entire heap only use 50% of RAM? Surely the -Xmx specifies the maximum RAM Java will use? – Oudh

..also, won't reducing the heap size increase the likelihood of an OOM? – Oudh

in the gist, the first line says that the jvm has invoked oom killer and the last but one line says that there was a out of memory :-) the 50% thing is more of a general guideline i use for myself rather than anything else, because the JVM needs space for natives (threads, primitives - basically runtime stacks), class loading and for its own operation. all of this is not accounted for in the heap. typically, the heap to other memory needs ratio is around 1:1 and hence the 50% rule (my own). OOM is also caused when the JVM is unable to create threads or load classes, so try reducing the heap – Pederson

Ah, apologies. When you said "OOM" I thought you meant a "Java OOM exception", not the OS complaining about an OOM – Oudh

Oh, I've reduced the heap to 15GB per your recommendation – Oudh

Answer 5

Ubuntu has a "watchdog" process which kills other processes when memory runs low. See the manpage: http://manpages.ubuntu.com/manpages/natty/man8/watchdog.8.html

Tabbatha