Is there any way to analyze a truncated Java Heap Dump (hprof file)?

Question

In my work, we are running into a difficult to reproduce OOM issue. Or, more accurately, it is very easy to reproduce on one system, making that system unusable, but difficult to reproduce anywhere else, given the same inputs.

The application is being run as a service using a service wrapper. We did manage to get the configuration changed to launch it with the option of outputting a heap dump file on OOM but, unfortunately, they were truncated, most likely due to the service wrapper timing out and killing the process as it wrote the file. This is readily apparent, since the max memory is set to 1GB, and the hprof files are as small as 700MB, which is too small to be the entire heap upon OOM.

It would take a lot of jumping through hoops to additionally configure the wrapper to give the java process a longer time to write out the heap, but we are pursuing this using these 2 options:

wrapper.jvm_exit.timeout=600
wrapper.shutdown.timeout=600

The question is, is there anything useful I can do with the truncated hprof files I have? Eclipse MAT chokes on them. Jhat appears to load them, but then only shows 3 instances of Java.Object of size 0 and nothing else. I tried YourKit and it couldn't write its oids file.

It seems to me like these files should have some useful, accessible information in them. Is there a tool that can read what's there?

Thank you for your time!

Peritoneum

For what it's worth here's a link to the "spec", described in hprof_b_spec.h. An hprof file has a small header followed by many records, so on a truncated file you could perhaps remove the last (incomplete) record, and salvage the rest, but even with that you'd get many records that would contain object IDs that would be declared in absent records... so probably unexploitable. – Schleswigholstein

Have you checked whether the hprof file is really truncated? The fact that Xmx=1g and the file is 700mb means nothing. – Seals

I've found the yourkit folks super responsive. have you posted anything on their forum? – Mistreat

@Peritoneum were you able to find a suitable tool for the job? – Snapper

No, I was not able to. We just got another dump file from a different environment that was not truncated. – Peritoneum

@OlegEstekhin The fact that we finally managed to reproduce the error on another system, with a longer wrapper timeout, and got a larger hprof file which could be analyzed, seems to imply it was. But, genuine curiosity, may I ask why Xmx=1g and the file being 700MB doesn't mean it's truncated? The heap dump should include the whole heap, and we knew it was an OOM error, so the heap should be 1GB... – Peritoneum

Answer 1

Best option for analyzing the dump file which i came across till date, is text editors like vim.

Referee

Answer 2

Use Jpofiler(https://www.ejtechnologies.com/products/jprofiler/overview.html). It's not free, but, it has a trial period.

The live memory and CPU view options are your best bet to isolate your issues. It generally runs reasonably well even on large dumps.

Grubb

How does this answer the question? Does JProfiler support loading truncated or otherwise corrupt heap dumps? The author states he already tried YourKit, which offers a "live" mode pretty similar to JProfiler. So this answer does not really help a lot in my opinion. – Catling

That did not work in my case: java.io.IOException: Could not read heap dump. It seems that the data is corrupted. at com.jprofiler.core.e.b.az(ejt:1452) at com.jprofiler.core.e.b.J(ejt:1029) at com.jprofiler.core.e.e.a.a(ejt:249) at com.jprofiler.core.e.b.i(ejt:500) at com.jprofiler.core.e.b.a(ejt:235) at com.jprofiler.core.e.a.run(ejt:108) at com.jprofiler.core.e.b.aE(ejt:250) – Hawkshaw