CompletableFuture / ForkJoinPool Set Class Loader

Question

I tackled down a very specific problem, whose solution seems to be something basic:

My (Spring) application's classloader hierarchy is something like this: SystemClassLoader -> PlatformClassLoader -> AppClassLoader

If I use Java CompleteableFuture to run threads. the ContextClassLoader of the threads is: SystemClassLoader -> PlatformClassLoader -> ThreadClassLoader

Thus, I cannot access any class in AppClassLoader although I have to because all external library classes reside there.

The source base is quite large so I don't want to/can't rewrite all the thread related pieces to something else (e.g. pass a custom executor to each call).

So my question is: How can I make the threads created by e.g. CompleteableFuture.supplyAsync() use the AppClassLoader as a parent? (instead of the PlatformClassloader)

I found out that ForkJoinPool is used to create the threads. But as it seems to me, everything there is static and final. So I doubt that even setting a custom ForkJoinWorkerThreadFactory with a system property will help in this case. Or would it?

Edit to answer the questions from the comments:

• where do you deploy to? Is this running within jetty / tomcat / any JEE container?

  • I'm using the default Spring Boot setup so an internal tomcat container is used.

• What is the exact issue you have?

  • The exact issue is: java.lang.IllegalArgumentException: org.keycloak.admin.client.resource.RealmsResource referenced from a method is not visible from class loader

• The jobs that you submit to supplyAsync() are created from the AppClassLoader, aren't they?

  • The supplyAsync is called from the MainThread which uses the AppClassLoader. But, debugging the applications shows that all such threads have PlatformClassLoader as their parent. As to my understanding, this happens because ForkJoinPool.commonPool() is constructed during the application startup (because it's static) and so uses the default class loader as the parent which is PlatformClassLoader. So, all threads from this pool get PlatformClassLoader as their parent for ContextClassLoader (instead of AppClassLoader).

  • When I'm creating my own executor inside the MainThread and pass this executor to supplyAsync everything works - and I can see during debugging that indeed now AppClassLoader is the parent of my ThreadClassLoader. Which seems to affirm my assumption in the first case that the common pool is not created by MainThread at least not when it's using AppClassLoader itself.

Full stacktrace:

java.lang.IllegalArgumentException: org.keycloak.admin.client.resource.RealmsResource referenced from a method is not visible from class loader
    at java.base/java.lang.reflect.Proxy$ProxyBuilder.ensureVisible(Proxy.java:851) ~[na:na]
    at java.base/java.lang.reflect.Proxy$ProxyBuilder.validateProxyInterfaces(Proxy.java:682) ~[na:na]
    at java.base/java.lang.reflect.Proxy$ProxyBuilder.<init>(Proxy.java:628) ~[na:na]
    at java.base/java.lang.reflect.Proxy.lambda$getProxyConstructor$1(Proxy.java:426) ~[na:na]
    at java.base/jdk.internal.loader.AbstractClassLoaderValue$Memoizer.get(AbstractClassLoaderValue.java:327) ~[na:na]
    at java.base/jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent(AbstractClassLoaderValue.java:203) ~[na:na]
    at java.base/java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:424) ~[na:na]
    at java.base/java.lang.reflect.Proxy.newProxyInstance(Proxy.java:999) ~[na:na]
    at org.jboss.resteasy.client.jaxrs.ProxyBuilder.proxy(ProxyBuilder.java:79) ~[resteasy-client-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.client.jaxrs.ProxyBuilder.build(ProxyBuilder.java:131) ~[resteasy-client-3.1.4.Final.jar!/:3.1.4.Final]
    at org.jboss.resteasy.client.jaxrs.internal.ClientWebTarget.proxy(ClientWebTarget.java:93) ~[resteasy-client-3.1.4.Final.jar!/:3.1.4.Final]
    at org.keycloak.admin.client.Keycloak.realms(Keycloak.java:114) ~[keycloak-admin-client-3.4.3.Final.jar!/:3.4.3.Final]
    at org.keycloak.admin.client.Keycloak.realm(Keycloak.java:118) ~[keycloak-admin-client-3.4.3.Final.jar!/:3.4.3.Final]

Intercellular

where do you deploy to? Is this running within jetty / tomcat / any JEE container? – Adrianople

What is the exact issue you have? The jobs that you submit to supplyAsync() are created from the AppClassLoader, aren't they? So they should have access to its classes. This seems like an XY problem to me. – Bowlder

Thank you both for your comments! Please find the answers in the edited question above. Thanks! – Injustice

You should provide the full stacktrace of your exception. Also, this might be something specific to keycloak – so the corresponding tag might be relevant. – Bowlder

I added the full stacktrace up to the point where I called the failing method. – Injustice

So that's exactly the same issue as you had in Spring Boot / Security classloader issues with Keycloak run from terminal. Did you solve that in the end? I also notice that this error message is specific to Java 9. Did you try in 8? It could be a regression. – Bowlder

Unfortunately, Java 8 is not an option. But thanks for the suggestions! And yes, this is the same issue. I will cross link the two questions. I found a solution for this one and will post it soon (though I'm not proud of the solution as it is really dirty). – Injustice

Answer 1

So, here is a very dirty solution of which I'm not proud of and may break things for you if you go along with it:

The problem was that the classloader of the application was not used for ForkJoinPool.commonPool(). Because the setup of commonPool is static and therefor during the application start up there is no easy possibility (at least to my knowledge) to make changes later. So we need to rely on Java reflection API.

1. create a hook after your application successfully started

   • in my case (Spring Boot environment) this will be the ApplicationReadyEvent

   • to listen to this event you need a component like the following

     @Component
     class ForkJoinCommonPoolFix : ApplicationListener<ApplicationReadyEvent> {
         override fun onApplicationEvent(event: ApplicationReadyEvent?) {
       }
     }
     

2. Inside your hook you need to set ForkJoinWorkerThreadFactory of commonPool to a custom implementation (so this custom implementation will use the app classloader)

   • in Kotlin

     val javaClass = ForkJoinPool.commonPool()::class.java
     val field = javaClass.getDeclaredField("factory")
     field.isAccessible = true
     val modifiers = field::class.java.getDeclaredField("modifiers")
     modifiers.isAccessible = true
     modifiers.setInt(field, field.modifiers and Modifier.FINAL.inv())
     field.set(ForkJoinPool.commonPool(), CustomForkJoinWorkerThreadFactory())
     field.isAccessible = false
     

3. Simple implementation of CustomForkJoinWorkerThreadFactory

   • in Kotlin

     //Custom class
     class CustomForkJoinWorkerThreadFactory : ForkJoinPool.ForkJoinWorkerThreadFactory {
       override fun newThread(pool: ForkJoinPool?): ForkJoinWorkerThread {
         return CustomForkJoinWorkerThread(pool)
       }
     }
     // helper class (probably only needed in kotlin)
     class CustomForkJoinWorkerThread(pool: ForkJoinPool?) : ForkJoinWorkerThread(pool)
     

If you need more information about reflection and why it's not good to change final fields please refer to here and here. Short summary: due to optimizations the updated final field may not be visible to other objects and other unknown side effects may occur.

As stated before: this is a very dirty solution. Unwanted side effects may occur if you use this solution. Using reflections like this is not a good idea. If you can use a solution without reflection (and post it as an answer here!).

Edit: Alternative for single calls

Like stated in the question itself: if you only have this problem in a small number of places (i.e. it's no problem to fix the call itself) you can use your own Executor. A simple example copied from here:

ExecutorService pool = Executors.newFixedThreadPool(10);
final CompletableFuture<String> future = 
    CompletableFuture.supplyAsync(() -> { /* ... */ }, pool);

Intercellular

Added a working solution without reflection. This should work for you as well given you are already hooking up a CustomForkJoinWorkerThreadFactory that has the correct class loader. – Brinkley

Answer 2

One possible solution valid in jdk11 (tested using Spring Boot 2.2) is to take advantage of the new constructors in ForkJoinPool

The main idea is to create a custom ForkJoinPool using a custom ThreadFactory that uses our own ClassLoader (not the system one -this behavior starts in jdk9-)

A bit of history
Before jdk9 ForkJoinPool.common() returns an Executor with a ClassLoader of your main Thread, in Java 9 this behave changes, and return an executor with the system jdk system classloader. So it's easy to find ClassNotFoundExceptions inside CompletableFutures code while upgrading from Java 8 to Java 9 / 10 / 11, due to this change.

Solution Create our own factory like Neo said in an anwser before and use this factory yo create a ForkJoinPool and an Executor

MyForkJoinWorkerThreadFactory factory = new MyForkJoinWorkerThreadFactory();

ForkJoinPool myCommonPool = new ForkJoinPool(Math.min(32767, Runtime.getRuntime().availableProcessors()), factory, null, false);

Use it like this

CompletableFuture.runAsync(() -> {
   log.info(Thread.currentThread().getName()+" "+Thread.currentThread().getContextClassLoader().toString());  
   // will print the classloader from the Main Thread, not the jdk system one :)
}, myCommonPool).join();

Extraball
If your are behind a Spring based app, should be necesary to add your Spring Security Context to the new custom thread pool

@Bean(name = "customExecutor")
public Executor customExecutor() {
    MyForkJoinWorkerThreadFactory factory = new MyForkJoinWorkerThreadFactory();
    ForkJoinPool myCommonPool = new ForkJoinPool(Math.min(32767, Runtime.getRuntime().availableProcessors()), factory, null, false);

    DelegatingSecurityContextExecutor delegatingExecutorCustom = new DelegatingSecurityContextExecutor(myCommonPool, SecurityContextHolder.getContext());
    return delegatingExecutorCustom;
}

And use it autowiring like any other resource

@Autowired private Executor customExecutor;

CompletableFuture.runAsync(() -> {
    ....
}, customExecutor).join();

Payment

Answer 3

I ran into something similar and came up with a solution that does not use reflection and seems to work well with JDK9-JDK11.

Here is what the javadocs say:

The parameters used to construct the common pool may be controlled by setting the following system properties:

• java.util.concurrent.ForkJoinPool.common.threadFactory - the class name of a ForkJoinPool.ForkJoinWorkerThreadFactory. The system class loader is used to load this class.

So if you rollout your own version of the ForkJoinWorkerThreadFactory and set that instead to use the correct ClassLoader using the system property, this should work.

Here is my custom ForkJoinWorkerThreadFactory:

package foo;

public class MyForkJoinWorkerThreadFactory implements ForkJoinWorkerThreadFactory {

    @Override
    public final ForkJoinWorkerThread newThread(ForkJoinPool pool) {
        return new MyForkJoinWorkerThread(pool);
    }

    private static class MyForkJoinWorkerThread extends ForkJoinWorkerThread {

        private MyForkJoinWorkerThread(final ForkJoinPool pool) {
            super(pool);
            // set the correct classloader here
            setContextClassLoader(Thread.currentThread().getContextClassLoader());
        }
    }
} 

and then set the system property in your app startup script

-Djava.util.concurrent.ForkJoinPool.common.threadFactory=foo.MyForkJoinWorkerThreadFactory

The above solution works assuming that when the ForkJoinPool class is referenced the first time and it initializes the commonPool, the context ClassLoader for this Thread is the correct one that you need (and is not the System class loader).

Here is some background that might help:

Fork/Join common pool threads return the system class loader as their thread context class loader.

In Java SE 9, threads that are part of the fork/join common pool will always return the system class loader as their thread context class loader. In previous releases, the thread context class loader may have been inherited from whatever thread causes the creation of the fork/join common pool thread, e.g. by submitting a task. An application cannot reliably depend on when, or how, threads are created by the fork/join common pool, and as such cannot reliably depend on a custom defined class loader to be set as the thread context class loader.

As a result of the above backward incompatibility change, things that uses the ForkJoinPool that used to worked in JDK8 may not work in JDK9+ .

Brinkley

This solution is not working in JDK9+, as you said, so I will post a new complete solution based on your solution below. Thanks – Sometimes

Shouldn't the classloader in MyForkJoinWorkerThread constructor be set like this? setContextClassLoader(MyForkJoinWorkerThreadFactory.class.getClassLoader()); – Lightweight

Answer 4

It seems that resteasy lib uses thread context classloader to load some resources: http://grepcode.com/file/repo1.maven.org/maven2/org.jboss.resteasy/resteasy-client/3.0-beta-1/org/jboss/resteasy/client/jaxrs/ProxyBuilder.java#21.

When resteasy try to load the requested class, it will ask the thread classloader to look for it and load it if possible, when the requested class is located in a classpath not visible by the classloader, the operation fails.

And that exactly what happens to your application: ThreadClassLoader tried to load a resource located in the application classpath, since resources from this classpath are accessible only from AppClassLoader and its children, then the ThreadClassLoader has failed to load it (ThreadClassLoader is not a child of AppClassLoader).

A possible solution could be to override the thread context ClassLoader by your app ClassLoader: thread.setContextClassLoader(appClass.class.getClassLoader())

Pollaiuolo

Thanks for your suggestion. This is basically what I already know but nevertheless I'm thankful for any constructive comment :) I upvoted your answer so you get some reputation for being able to post comments ;) – Injustice

thanks for the up-vote :) yep, I wanted to post it as a comment, but since I cannot I was obliged to put it as an answer. – Pollaiuolo