Java memory leak - jmap doesn't show the classes but jstat does

Question

I troubleshoot a weird memory leak. It's specific to Java8, not happening on 7u79.

I don't have access to Java code. I know exactly which user action causes a leak, I know the leak is about the classes (not the heap) and the offending classes are easy to spot with +TraceClassLoading +TraceClassUnloading:

[Loaded com.mastercard.mcwallet.sdk.xml.allservices.ShoppingCartRequest$JaxbAccessorF_oAuthToken from __JVM_DefineClass__]
[Loaded com.mastercard.mcwallet.sdk.... thousand similar classes per one user action... ]

These classes seem to increase the class counter output by jstat -class:

Loaded  Bytes  Unloaded  Bytes     Time
 14045 26138.8        0     0.0     110.00   << buggy user action
 14675 26754.6        0     0.0     110.05
 15300 27364.9        0     0.0     110.10
 15304 27370.9        0     0.0     110.11
 15304 27370.9        0     0.0     110.11
 15304 27370.9        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11
 15306 27374.0        0     0.0     110.11   << buggy user action
 15930 27982.2        0     0.0     110.18
 16553 28589.3        0     0.0     110.23
 16553 28589.3        0     0.0     110.23

The thing is that these classes are never garbage-collected from metaspace, never [Unloaded] and they don't show in jmap -clstats. The command reports a lower number of classes, the number doesn't increase, there are no suspicious class loaders:

class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     2574    4493256   null          live    <internal>
0x0000000087d016d0      1       1471    0x000000008237f088      dead    sun/reflect/DelegatingClassLoader@0x00007ff4135d02d0
... some lines omitted ...
0x000000008237f088      6505    12228227        0x0000000080383938      dead    org/apache/catalina/loader/WebappClassLoader@0x00007ff411546ad0
... some lines omitted ...
total = 600     14002   25351427            N/A         alive=1, dead=599           N/A

Does this ring any bell or brings any tips I could pass to programmers? They say they are unable to locate the leak. Can I stop this leak just by fiddling with JVM options?

Darryldarryn

The problem only with MetaSpace? what about heap? and can you also provide the Java process memory usage statistic? – Chairmanship

Only Metaspace, heap is stable and not full. What statistic exactly? – Darryldarryn

I mean how fast Java process memory usage grows? – Chairmanship

Each "buggy user action" allocates about 2 MB from OS (pmap tool even shows a separate 2 MB segment). – Darryldarryn

have you tried to use different GC algorithms, for example G1 -XX:+UseG1GC? Or you can try to set -XX:MaxMetaspaceSize=<NNN> to expectedly define the size of Metaspace. – Chairmanship

Yup, tried them all, no difference at all. These classes seem simply not eligible for GC in Java8, but somehow can disappear on Java7. With MaxMetaspaceSize the app simply dies as soon as it fills up. – Darryldarryn

You can try to use Eclipse Memory Analyzer ( MAT ) to inspect your heap dump. If buggy classes unavaliable for unloading then it must be cached somewhere in application, and MAT can show you where it is. – Chairmanship

Answer 1

I presume JaxB is loading classes into your memory, which are not being garbage collected. Metaspace will not empty out if the Classes are not unloaded.

Check for JaxB memory issue for Java 8. I think you can get rid of this exception by swapping the dependency api version or by toggling on an extra parameter:

-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

Noni

Answer 2

If buggy classes unavailable for unloading then it must be cached somewhere in application, and as I understand your goal is to find where it is and provide this information for developers.

You can try to use Eclipse Memory Analyzer (MAT) to inspect your heap dump, and it can show problem areas, classes and instances which has links to the buggy classes.

Chairmanship