Old JaxB and JDK8 Metaspace OutOfMemory Issue

Question

We are working on a business application (1 million+ LOC) developed since 10+ years. While switching to JDK8 we get an issue with the metaspace of JDK8. This seems to be related to the JaxB-Version referenced in com.sun.xml.ws:webservices-rt:1.4 (Metro 1.4). Because of the intense linking in the application and legacy creation of classes/instances via JaxB it isn't simple to switch on the fly the old libraries.

Currently we are researching this issue. We created a sample programm that reproduces this behavior:

import java.io.ByteArrayInputStream;

import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Unmarshaller;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class X
{
  private static final String XML = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><x test=\"test\" />";

  @XmlAttribute
  String test;

  public static void main( String[] args ) throws JAXBException, InterruptedException
  {
    System.out.println("start");

    while ( true )
    {
      JAXBContext jc = JAXBContext.newInstance( X.class );
      Unmarshaller unmarshaller = jc.createUnmarshaller();
      X object = (X) unmarshaller.unmarshal( new ByteArrayInputStream( XML.getBytes() ) );
      System.out.println( object.test );
    }
  }
}

JDK7 keeps the PermGenSpace clean. (Simulated with 16M PermGen) Memory of run with JDK7

Using JDK8 the application runs slowly to the OOM exception. The VisualVM catches the exception and keeps the process running on the maximum of available Metaspace. Even here it gets stucked after quite a while running on max. (Simulated with 16M Metaspace) Memory of run with JDK8

Has anyone some ideas how to get the garbage collectors legacy behavior, so we don't run into those out of memory issues? Or do you have any other ideas how to deal with this issue?

Thanks.

edit1: Run parameters JDK7:

-XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:MaxPermSize=16M -XX:PermSize=1M -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

=> No heap dumps are created

Run parameters JDK8:

-XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:MaxMetaspaceSize=16M -XX:MetaspaceSize=1M -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

=> heap dumps are generated while running.

The Memory available of VisualVM does not show the real maximum metaspace value. If not limited the metaspace is constantly increasing untill memory is exceeded.

edit 2:

I have tried all available garbage collectors for JDK8. They all have the same issue.

edit 3:

Solving by exchanging the libs is difficult in our real application because of heavy coupling between JAXB & several modules of our application. So a fix for the garbage collector behavior is needed for the short run. On the long run the propper fix is already planned.

Paperboy

“VisualVM catches the exception and keeps the process running”—I never knew that VisualVM can do that. Besides that, your screenshot says that in your Java8 run, the Metaspace has a limit of 1GB but stays at 16MB, which is far away from running towards an OOME. The Metaspace doesn’t seem to get collected, but there also seems to be a simple, obvious reason: in the Java 8 setup, it keeps the classes as the counter stays constantly at ~4300 classes, whereas in the Java 7 run, a whopping number of ~43,000 classes have been counted. So there, classes seem to get constantly regenerated… – Fording

The OOMs are shown in the Eclipse debugger, while running. The Task is limited by parameters on the run configuration to 16MB Metaspace. This does not apply to the VisualVM instance. So this says other amounts of memory left. If I leave the application running without VisualVM the OOM stops the process at right the same moment the JDK8 process hits 16MB Metaspace. When using e.g. webservices-rt 1.6.1 or removing this and using the JDK8 JaxB-implementation the sample programm keeps a constant amount of 12.5MB Metaspace. If not limiting JDK8 Memory it will increase untill all is used. – Paperboy

I’m confused. What do the screenshots show? Are you analyzing your application or VisualVM? – Fording

The Pictures show the analysis of the run of the sample programm. I have added the run parameters in the original article. It seems the VisualVM does not show the correct MaxMetaspace value. java.net/jira/browse/VISUALVM-609 – Paperboy

Sorry but your explanation is still confusing. Especially the sentence “When using e.g. webservices-rt 1.6.1 or removing this and using the JDK8 JaxB-implementation the sample programm keeps a constant amount of 12.5MB Metaspace.” How does this setup differ from the follow-up “If not limiting JDK8 Memory it will increase untill all is used”? And are you now talking about “Memory” or “Meta-Space”? – Fording

We are talking on PermGen- (JDK7) and on MetaSpace (JDK8).Like shown above in the post the memory partition is limited to 16MB of memory (PermGen/MetaSpace) to provoke a fast establishing of our issue.Using newer JaxB libs with the same config keeps the memory within this thin borders and not requiring additional Space per iteration of the loop.On our systems this is used on some timed processes with the effect that JDK7 works perfectly with less than 2GB(whole service) available RAM and that using JDK8 the same system needs 6+GB RAM after some days of running!The sample has the same behavior. – Paperboy

First of all, PermGen and MetaSpace are different things and it is wrong to assume, limiting both to the same size was doing the same thing. Second, if you already know that “using newer JaxB libs” solves your problem, then you already know where’s the issue and how to solve it. If the old libraries have a memory leak, replace them instead of playing with JVM options… – Fording

The memory leak only occurs when using Java 8, in case of Java 7 the GC cleans unused classes without problems. As stated in the question switching the JaxB version is only viable in the long run for us. – Paperboy

Answer 1

We solved our current issue untill able to fix all occurances in our application by using the following VM-parameter:

-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

I hope this will help others with similar issues...

Paperboy

We're also facing similar issue related to OOME when we switched our app from JDK7 to JDK8 and this parameter solved our problem. Thank you for posting the solution !!! – Schlessel

Thanks, that helped. You also have the option to put the JAXBContext.newInstance outside the loop (if it is in your code) and keep it as a singleton. Also, the bug seems to have been fixed with jaxb-2.1.13 (jira ticket : JAXB-564) – Chante

Thanks, this solved the same problem with JAX-WS client, which use JAXB for data binding. – Cheremkhovo

To ease the search: JAXB-564 – Tingle

in which file need to put? – Villainy

Answer 2

Here is the solution Gary is talking about, which is better than just setting a flag (since even the JAXB guys suggest to make it singleton...)

private static Map<class<?>, JAXBContext> contextStore = new ConcurrentHashMap<class<?>, JAXBContext>();
... 
protected static JAXBContext getContextInstance(Class<?> objectClass) throws JAXBException{
  JAXBContext context = contextStore.get(objectClass);
  if (context==null){
    context = JAXBContext.newInstance(objectClass);
    contextStore.put(objectClass, context);
  }
  return context;
}

//using it like this:
JAXBContext context = getContextInstance(objectClass);

The JAXB-564 Bug + fix could be found here. Inspired by the long gone blog of scorgar

Tingle

inspired by scorgar.be/blog/… – Tingle

Yes but it is available forever here: web.archive.org/web/20171228110409/http://www.scorgar.be:80/… – Tingle

Answer 3

JAXBContext.newInstance() should be used once to create a context for your class to unmarshall. It will use up your permgen or metaspace otherwise.

Cambrai