Hazelcast on deployed servers throws java.lang.ClassNotFoundException and locally not

Question

I wanted to upgrade to Java 11 and tomcat 9 my spring boot application that uses Hazelcast 3.12.9 as cashing mechanism. When I deployed locally everything looks to work fine and the caching successfully works. But when the application runs on the cluster, I receive from all 3 nodes that are available the following error:

com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: com.some.service.some.server.domain.ClassA
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:88)
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77)
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
    at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1237)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)
    at com.hazelcast.spring.cache.HazelcastCache.lookup(HazelcastCache.java:162)
    at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:67)
    at com.some.service.some.server.domain.ClassACache.get(GlassACache.java:28)
    at com.some.service.some.server.domain.ClassAFacade.getClassA(ClassAFacade.java:203)
    at com.some.service.some.server.domain.ClassAFacade.getGlassA(ClassAFacade.java:185)
    at com.some.service.some.server.domain.ClassALogic.lambda$getClassAInParallel$1(ClassALogic.java:196)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:952)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:926)
    at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.ClassNotFoundException: com.some.service.some.server.domain.ClassA
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)

Hazelcast customizer :

@Configuration
public class ClassAHazelcastConfig {
    private static final MaxSizePolicy HAZELCAST_DEFAULT_MAX_SIZE_POLICY = MaxSizePolicy.PER_NODE;
    private static final EvictionPolicy HAZELCAST_DEFAULT_EVICTION_POLICY = EvictionPolicy.LRU;

    @Bean
    HazelcastConfigurationCustomizer customizer(CachePropertiesHolder cacheProperties) {
        return config -> {
            config.addMapConfig(new MapConfig()
                    .setName(CLASS_A_CACHE)
                    .setMaxSizeConfig(new MaxSizeConfig(cacheProperties.getMaxsize(), HAZELCAST_DEFAULT_MAX_SIZE_POLICY))
                    .setEvictionPolicy(HAZELCAST_DEFAULT_EVICTION_POLICY)
                    .setTimeToLiveSeconds(cacheProperties.getTtl()));

            config.getSerializationConfig().addSerializerConfig(
                    new SerializerConfig()
                            .setImplementation(new OptionalStreamSerializer())
                            .setTypeClass(Optional.class)
            );
        };
    }
}

@Configuration
@EnableConfigurationProperties(CachePropertiesHolder.class)
public class CacheConfig implements CachingConfigurer, EnvironmentAware, ApplicationContextAware {

    public static final String CLASS_A_CACHE = "CACHE_A";

    private Environment environment;
    private ApplicationContext applicationContext;

    @Override
    @Bean(name="cacheManager")
    public CacheManager cacheManager() {
        boolean cachingEnabled = Boolean.parseBoolean(environment.getProperty("cache.enabled"));
        if (cachingEnabled) {
            HazelcastInstance instance = (HazelcastInstance) applicationContext.getBean("hazelcastInstance");
            return new HazelcastCacheManager(instance);
        }
        return new NoOpCacheManager();
    }

    @Override
    public CacheResolver cacheResolver() {
        return new SimpleCacheResolver(Objects.requireNonNull(cacheManager()));
    }

    @Bean
    @Override
    public KeyGenerator keyGenerator() {
        return new SimpleKeyGenerator();
    }

    @Bean
    @Override
    public CacheErrorHandler errorHandler() {
        return new SimpleCacheErrorHandler();
    }

    @Override
    public void setEnvironment(@NotNull Environment environment) {
        this.environment = environment;
    }

    @Override
    public void setApplicationContext(@NotNull ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }
}

Everything works properly fine with Java 8 and tomcat 8.

Update:

After some days of investigation, I see that the only place that these exceptions are thrown into a parallel stream that is used.

return forkJoinPool.submit(() ->
                    items.parallelStream()
                            .map(item -> {
                                try {
                                    return biFunction.apply(item);
                                } catch (Exception e) {
                                    LOG.error("Error", e);
                                    return Optional.<Item>empty();
                                }
                            })

The weird thing is that with Java 8 and tomcat 8 I did not have that issue.

Glanti

Please share your code base through github . – Responsiveness

Unfortunately, It is internal code and I cannot do that. – Glanti

Search through the whole project for ClassA. If it exists, then you won't get this error. – Responsiveness

It for sure exists and it is part of the basic project. – Glanti

You need to upgrade HazelCast as well. Latest version is 4.1.1 – Responsiveness

Have you reviewed at this? #22527889 @Glanti – Ingemar

@AnishB. is there something that has changed in that version relevant to this or just upgrade to check? – Glanti

You need to check on that. – Responsiveness

@iswailyildiz I just checked it but apparently, in my case, I do not explicitly define any format or similar things – Glanti

Answer 1

Eventually, it is completely unrelated to Hazelcast. Mostly it is Java 11 difference from Java 8.

In the part that the exception was thrown a ForkJoinPool was used which from Java 11, it is not guaranteed when exactly will be created and it seems to not have the same classloader as the spring application. (Classes with default access result in NoClassDefFound error at runtime in spring boot project for java 11)

I made a wrong assumption because the exception was coming from Hazelcast and I also saw that there were other issues related.

Glanti

Answer 2

Hazelcast offers two deployment models: embedded mode and client-server mode. Please check the relevant documentation section for more information.

In the former case, you have a single JVM and all of your classes are on your classpath.

In the latter case, you have at least 2 JVMs, one for each client and one for the member. It seems you forgot to set up the classpath of the member to reference ClassA.

How to set the classpath depends on how you launch your Hazelcast members. The method that works in most cases is to use the CLASSPATH environment variable.

Stambaugh

Thank you very much Nicolas for your answer. I use the embedded deployment model but I am curious why the behavior of the application is different with the upgrade of java from 8 to 11 and tomcat from 8 to 9. – Glanti

I'm pretty sure the behavior is not different. Perhaps you had user-code deployment in the past? Impossible to know for sure. – Stambaugh

@PavlMits, please how can I reach you as I am having this same problem but am not using spring boot but traditional spring. – Bouffe