Performance drop upgrading from Hazelcast 2.5 to 3+

Question

Because of a known (fixed) bug in Hazelcast 2.5 we've decided this would be the next upgrade candidate for our project. But after dropping in the latest version (3.2.2) we had horrible performance.

The way we are using Hazelcast:

• Two nodes
• Multiple IMap instances (about 7 maps in total)
• Both nodes update the maps
• A lot of reads on the maps
• near-cache enabled to speed up reads

Using Hazelcast 2.5 we had great performance when, instead of using map.values(), we supplied a list of all contained keys map.getAll(containedKeys). The way we keep track of the containedKeys by adding an EntryListener to the map which stores the containedKeys in a concurrent set. This was added by a colleague and feels like a hack, but works like a charm.

Now when we upgrade to Hazelcast 3.2.2 we instantly see problems with java.io, for example look at the following snippet from AppDynamics:

com.hazelcast.map.proxy.MapProxyImpl:getAll:326 (method time = 0 ms, total time = 18938 ms)
 com.hazelcast.map.proxy.MapProxySupport:getAllObjectInternal:495 (method time = 0 ms, total time = 18938 ms)
  com.hazelcast.map.MapService:toObject:852 (method time = 0 ms, total time = 18938 ms)
   com.hazelcast.spi.impl.NodeEngineImpl:toObject:156 (method time = 0 ms, total time = 18938 ms)
    com.hazelcast.nio.serialization.SerializationServiceImpl:toObject:221 (method time = 0 ms, total time = 18938 ms)
     com.hazelcast.nio.serialization.StreamSerializerAdapter:read:59 (method time = 0 ms, total time = 18938 ms)
      com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer:read:185 (method time = 0 ms, total time = 18938 ms)
       java.io.ObjectInputStream:readObject:370 (method time = 3398 ms, total time = 18938 ms)
        java.io.ObjectInputStream:readObject:370 (method time = 15540 ms, total time = 15540 ms)

This is something we haven't seen in Hazelcast 2.5, but do have in 3.2.2. It grinds our application to a complete standstill. Replacing the jar with 2.5 again (and renaming Entry back to MapEntry) and nothing is wrong.

What could be causing this? Maybe it isn't using the near-cache anymore?

Unanswerable

Is there a chance to quickly build your own snapshot and try it? I know there was a nearcache bug but not sure if it applies to this situation. – Kalimantan

I've already patched this: github.com/hazelcast/hazelcast/pull/2523 it fixed the loadClass but this readObject is now the bottleneck. Please point me to a possible patch/direction. – Unanswerable

Well Java standard serialization will always be the bottleneck. If you want high speed just don't use it or at least use j.i.Externalizable. – Kalimantan

@RoyvanRijn: Hazelcast's codebase is still undergoing an alarming amount of code churn. We've seen wild swings in performance going from one minor version to another. My advice would be to stick with the specific version that works best for you, and try and get your bug fixed in a point release of that. – Gifted

You are making use of Java Serializable. I have seen some issues with that; for a quick check you might switching to a different serializer and see if that gives a performance boost. – Refluent

Answer 1

did you take a look to this google group ticket issue?

https://groups.google.com/forum/#!topic/hazelcast/ivk6hzk2YwA

Here in particular looks the reason of the issue.

https://github.com/hazelcast/hazelcast/issues/553

Olenta