Springboot embedded Tomcat classloader slowness

Question

I have built a web application that uses SpringBoot v1.3.6.RELEASE Tomcat 8.0.36 Java 1.8u101 on CentOS 7.2

The web application is also a SOAP client that calls out to another web application.(JAX-WS RI 2.2.9) If the applications remains idle for 15 seconds the first webservice call stalls for nearly 2 seconds. It appears that the stall happens in o.a.c.loader.WebappClassLoaderBase.

After idle 15 seconds

16:02:36.165 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader@45283ce2

16:02:36.170 : Searching local repositories

16:02:36.170 : findResource(META-INF/services/javax.xml.soap.MetaFactory)

16:02:38.533 : --> Resource not found, returning null

16:02:38.533 : --> Resource not found, returning null

Next request no idle time

16:07:09.981 : Delegating to parent classloader org.springframework.boot.loader.LaunchedURLClassLoader@45283ce2

16:07:09.984 : Searching local repositories

16:07:09.985 : findResource(META-INF/services/javax.xml.soap.MetaFactory)

16:07:09.986 : --> Resource not found, returning null

16:07:09.986 : --> Resource not found, returning null

16:07:09.988 : findResources(META-INF/services

All above messages produced by o.a.c.loader.WebappClassLoaderBase and they are apparently being caused by ClientSOAPHandlerTube.processRequest which is from JAX-WS RI.

You'll notice the first call takes over 2 seconds but subsequent calls take only milliseconds. I'm wondering if anyone has experienced this behavior?

Possible solutions: Is it possible to change out the classloader used by tomcat in springboot to use ParallelWebappClassLoader

Or maybe this is a product of the reloadable flag on the classloader but I don't see how to change that flag in springboot.

When run using Jetty as the container this does not occur.

Final Solution: (thanks to Gergely Bacso)

    @Bean
public EmbeddedServletContainerCustomizer servletContainerCustomizer() {
    return new EmbeddedServletContainerCustomizer() {

        @Override
        public void customize(ConfigurableEmbeddedServletContainer container) {
             if (container instanceof TomcatEmbeddedServletContainerFactory) {
                customizeTomcat((TomcatEmbeddedServletContainerFactory) container);
            }
        }
        private void customizeTomcat(TomcatEmbeddedServletContainerFactory tomcatEmbeddedServletContainerFactory) {
            tomcatEmbeddedServletContainerFactory.addContextCustomizers(new TomcatContextCustomizer() {
                @Override
                public void customize(Context cntxt) {
                    cntxt.setReloadable(false);
                }
            });
        }
    };
}

Esbenshade

It looks like there's a cache that's getting invalidated by the 15 seconds of idle time so the findResource call has to search the entire classpath again. I can't tell where that cache might be as it's not clear what's generating the output that you've shared above. A complete example that reproduces the problem would help here. I'd also take a look at whatever's looking for the META-INF/services/javax.xml.soap.MetaFactory resource. Performing that look up for every request seems unnecessary as the result's very unlikely to have changed. – Gagliardi

@AndyWilkinson yes that is my thoughts exactly. I have looked at WebappClassLoaderBase and I don't see any cache's. Also I agree with you that a lookup every request seems wasteful. Maybe I will try CXF instead of Metro and see if the same thing happens. – Esbenshade

I came across this Q after profiling some drastic class loader performance issues during a migration to add Spring Boot and embedded Tomcat. reloadable seems to be set to false by default now, but, for anyone else running across this, there is a similar class loader issue which only applies when packaging the application as a WAR: github.com/spring-projects/spring-boot/issues/16471. A suggested solution on that ticket fixed it for me. – Ouellette

Answer 1

Actually your findings are quite good and you have 90% answered your question already. These two facts:

1. "it appears that the stall happens in o.a.c.loader.WebappClassLoaderBase"
2. "when run using Jetty as the container this does not occur."

show that it is going be a Tomcat-related problem because:

1. o.a.c. stands for org.apache.catalina
2. Your code works well on another container. (Jetty)

You also observed, that the issue is happening after 15 seconds of idle time. This perfectly corresponds to Tomcat's default checkInterval setting, which is:

The number of seconds between checks for modified classes and resources, if reloadable has been set to true. The default is 15 seconds.

So in short: currently your reloadable flag is ON, and Tomcat tries to reload your classes which is handy during development, but unacceptable in any other case. The way to switch it off is not via Spring-boot though.

SOLUTION:
You need to locate your context.xml / server.xml where you will find your Context defined like this:

<Context ... reloadable="true">

Remove the reloadable flag, and you have solved the problem. The file itself can be either in $CATALINA_BASE/conf of $CATALINE_HOME/conf, but in reality these locations can be a bit tricky to find if you are using some IDE to manage Tomcat for you.

In case of embedded Tomcat with Spring-boot:

The class you can use to manipulate Tomcat settings is: EmbeddedServletContainerCustomizer.

Through this you can add a TomcatContextCustomizer (addContextCustomizers) so that you can call setReloadable on the context itself.

I do not see any reason for Spring-boot needing this flag on true.

Tawnytawnya

That flag must be set to true by default in springboot's embedded tomcat.. In a springboot application the tomcat is embedded in the application jar file and as far as I can tell it does not use any context.xml file. I will continue to dig on how it is possible to change that flag in a springboot application – Esbenshade

@cyberoblivion, I see the problem. I expanded the answer. – Tawnytawnya

Basco, I updated the question with the solution I came up with based on your feedback! thanks. – Esbenshade