Why is my Java heap dump size much smaller than used memory?

Question

Problem

We are trying to find the culprit of a big memory leak in our web application. We have pretty limited experience with finding a memory leak, but we found out how to make a java heap dump using jmap and analyze it in Eclipse MAT.

However, with our application using 56/60GB memory, the heap dump is only 16GB in size and is even less in Eclipse MAT.

Context

Our server uses Wildfly 8.2.0 on Ubuntu 14.04 for our java application, whose process uses 95% of the available memory. When making the dump, our buffers/cache used space was at 56GB.

We used the following command to create the dump: sudo -u {application user} jmap -dump:file=/mnt/heapdump/dump_prd.bin {pid}

The heap dump file size is 16,4GB and when analyzing it with Eclipse MAT, it says there are around 1GB live objects and ~14,8GB unreachable/shallow heap.

EDIT: Here is some more info about the problem we see happening. We monitor our memory usage, and we see it grow and grow, until there is ~300mb free memory left. Then it stays around that amount of memory, until the process crashes, unfortunately without error in the application log.

This makes us assume it is a hard OOM error because this only happens when the memory is near-depleted. We use the settings -Xms25000m -Xmx40000m for our JVM.

Question

Basically, we are wondering why the majority of our memory isn't captured in this dump. The top retained size classes don't look too suspicious, so we are wondering if there is something heap dump-related what we are doing wrong.

Wolsey

How are you measuring the application memory usage? Just because the Java process uses X amount of memory, doesn't mean that the Java heap is X. – Winy

Good one, we are using the linux command free -h to see what our memory usage is. – Wolsey

"until the process crashes, unfortunately without error in the application log" - check out the directory with the server executables; that is generally the directory where the 'java' command is invoked and will be the directory where the JVM will create a crash report file. See if such files exist, it may give a clue. What you describe sounds like a hard virtual machine crash, not a normal Java application exception. – Ashtray

Are you capturing stdout and stderr from the JVM? If not, try redirecting those to a file and you might see the exception there. – Mello

Thanks for the suggestions Gimby & schtever, much appreciated! – Wolsey

@Thermometer, what is the cause and reason, did you finally find out? I have nearly the same problem. – Cuddy

@Jaskey you mean the cause & reason of the heap dump size? Or the cause & reason of the OOM error? – Wolsey

@Wolsey Cause of why the RES memory is much larger than the heap size dumped. I suffered from the same problem and looking for an answer. My program should only consume around 3.9 g memory since the dump size is 3.9g but the RES size is much larger until it reached the xms value. – Cuddy

@Jaskey in my case, what happened was that my program didn't have enough time to do a full garbage collection, so memory kept building. When doing the dump, it had enough time to do a GC, which frees up the memory that was building up. That is why I was seeing the difference between heap dump size and memory usage. Pretty much what the accepted answer says. – Wolsey

@Thermometer, but I my case, I do not have a busy program, and even after I dump the heap, the RES is still the same large. I wonder if the actual RES is always larger because of the scale of jvm heap ? – Cuddy

Answer 1

When dumping its heap, the JVM will first run a garbage collection cycle to free any unreachable objects.

How can I take a heap dump on Java 5 without garbage collecting first?

In my experience, in a true OutOfMemoryError where your application is simply demanding more heap space than is available, this GC is a fool's errand and the final heap dump will be the size of the max. heap size.

When the heap dump is much smaller, that means the system was not truly out of memory, but perhaps had memory pressure. For example, there is the java.lang.OutOfMemoryError: GC overhead limit exceeded error, which means that the JVM may have been able to free enough memory to service some new allocation request, but it had to spend too much time collecting garbage.

It's also possible that you don't have a memory problem. What makes you think you do? You didn't mention anything about heap usage or an OutOfMemoryError. You've only mentioned the JVM's memory footprint on the operating system.

Disrespectable

Very nice explanation! i have updated the question with the answers to the things you ask. This could be the reason the heap dump is so much lower in size. – Wolsey

Unfortunately this is not exactly true. This only happens when the dump operation is used with the live sub-option : jmap -dump:live,file=... which in this case the there will be a full GC in order to dump only live objects. Which precisely did not happen there was both the command and mat indicate unreachable objects made it in the dump. There's several reasons why this could happen ful GC sometime before the dump, wide pointers... – Ahithophel

Answer 2

In my experience, having a heap dump much smaller than the real memory used can be due to a leak in the JNI.

Despite you don't use directly any native code, there are certain libraries that use it to speed up.

In our case, it was a Deflater and Inflater not properly ended.

Mitzvah