ClassCastException with Stacktrace Hazelcast version 4.2.5 using ReplicatedMap

Question

Using Hazelcast version 4.2.5 in a webapp deployed on Tomcat on Kubernetes. We're frequently("every 5 seconds") seeing ClassCastException with a stacktrace in the application logs.

Here's the ClassCastException :

java.lang.ClassCastException: class java.lang.String cannot be cast to class com.hazelcast.internal.serialization.impl.HeapData (java.lang.String is in module java.base of loader 'bootstrap'; com.hazelcast.internal.serialization.impl.HeapData is in unnamed module of loader org.apache.catalina.loader.ParallelWebappClassLoader @2f04993d)
27-Oct-2022 22:57:56.357 WARNING [hz.rogueUsers.cached.thread-2] com.hazelcast.internal.metrics.impl.MetricsCollectionCycle.null Collecting metrics from source com.hazelcast.replicatedmap.impl.ReplicatedMapService failed
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
        at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77)
        at com.hazelcast.internal.metrics.impl.MetricsService.collectMetrics(MetricsService.java:154)
        at com.hazelcast.internal.metrics.impl.MetricsService.collectMetrics(MetricsService.java:160)
        at com.hazelcast.internal.metrics.impl.MetricsRegistryImpl.collect(MetricsRegistryImpl.java:316)
        at com.hazelcast.internal.metrics.impl.MetricsCollectionCycle.collectDynamicMetrics(MetricsCollectionCycle.java:88)
        at com.hazelcast.replicatedmap.impl.ReplicatedMapService.provideDynamicMetrics(ReplicatedMapService.java:387)
        at com.hazelcast.replicatedmap.impl.ReplicatedMapService.getStats(ReplicatedMapService.java:357)
        at com.hazelcast.replicatedmap.impl.ReplicatedMapService.getLocalReplicatedMapStats(ReplicatedMapService.java:197)
        at com.hazelcast.replicatedmap.impl.LocalReplicatedMapStatsProvider.getLocalReplicatedMapStats(LocalReplicatedMapStatsProvider.java:85)

Here's how we're setting up Hazelcast.

    private static HazelcastInstance setupHazelcastConfig() {
        Config config = new Config();
        config.setInstanceName("rogueUsers");
        NetworkConfig network = config.getNetworkConfig();
        network.setPort(5701).setPortCount(20);
        network.setPortAutoIncrement(true);
        JoinConfig join = network.getJoin();
        join.getMulticastConfig().setEnabled(true);
//        join.getTcpIpConfig()
//        .setEnabled(true);

        HazelcastInstance hz = Hazelcast.getOrCreateHazelcastInstance(config);

        ReplicatedMapConfig replicatedMapConfig =
                config.getReplicatedMapConfig("rogueUsers");

        replicatedMapConfig.setInMemoryFormat(InMemoryFormat.BINARY);
        replicatedMapConfig.setAsyncFillup(true);
        replicatedMapConfig.setStatisticsEnabled(true);
        replicatedMapConfig.setSplitBrainProtectionName("splitbrainprotection-name");

        ReplicatedMap<String, String> map = hz.getReplicatedMap("rogueUsers");
        map.addEntryListener(new RogueEntryListener());

        return hz;
    }

Is this a configuration issue ?
How do I fix this ?

Thanks very much,

Aden

please add the full stack trace with "caused by". – Nels

@Nels : Unfortunately, the logs only contain the just this much. It repeats the same chunk of text after every 5 seconds. It doesn't have any cause. – Aden

Hey @Aden did my answer provide a solution to your problem? If so I would really appreciate it if you chose it as the response... – Caecum

Answer 1

The exception is being thrown from the following line:

if (isBinary) {
  memoryUsage += ((HeapData) record.getValueInternal()).getHeapCost(); <-- exception
}

which is line 85 of com.hazelcast.replicatedmap.impl.LocalReplicatedMapStats class. The condition being checked is as the following:

boolean isBinary = (replicatedMapConfig.getInMemoryFormat() == InMemoryFormat.BINARY);

so basically, it is related to the format you are saving the data (from the config above you have chosen BINARY).

However, I don't think you are following it correctly since you do the following: ReplicatedMap<String, String> map = hz.getReplicatedMap("rogueUsers"); in your config.

From the Javadoc of com.hazelcast.internal.serialization.Data class:

Data is basic unit of serialization. It stores binary form of an object serialized by SerializationService.toData(Object).

Therefore, try editing your config to this:

ReplicatedMap<Data, Data> map = hz.getReplicatedMap("rogueUsers");

Caecum

Depending on your configuration of what keys you use for data saving it might be: ReplicatedMap<String, Data> map = hz.getReplicatedMap("rogueUsers"); – Caecum