Hazelcast threads prevent TomEE from stopping

Question

Context

We want to use Hazelcast as our JCache implementation inside TomEE. As we don't need insane performance, at the moment, we want to run the Hazelcast node as part of our application.

We use Hazelcast 3.7 and TomEE 7.0.1.

Problem

When stopping TomEE, it complains about WARNING - The web application [APPLICATION_NAME] appears to have started a thread named [SOMENAME] but has failed to stop it. This is very likely to create a memory leak. several times and the VM will not halt normally but keep running.

The workaround obviously is to kill the process as soon as it looks idle. Needless to say that this is driving our developers and dev ops crazy.

Separate Hazelcast node

To rule out the possibility that the problems arise from the Hazelcast node being run inside TomEE, I tried to start a stand alone Hazelcast node and changed our application to only use the Hazelcast client to connect to said node. The behaviour stayed the same. As far as I can tell from several web searches, the Hazelcast client starts several threads as well, to communicate with the server nodes.

No duplicate of Hazelcast prevents the JVM from terminating

This question is not a duplicate of Hazelcast prevents the JVM from terminating as we totally rely on Hazelcasts JCache implementation. We do not access the Hazelcast instance directly and thus, we cannot call shutDownAll().

Test case

I've created a small test case on GitHub to reproduce the problem.

Questions

• Can we use Hazelcast as JCache backend in a Java EE application?
• What do we have to do to allow the application to stop normally?
• Can we also run the Hazelcast node as part of our application? If not: Why is this a bad idea?

Benjie

Possible duplicate of Hazelcast prevents the JVM from terminating – Stamford

Answer 1

Hazelcast uses its own threads and they are not always daemon, you can ensure you shutdown your hazelcast instance (client or node) through a producer like the one in https://issues.apache.org/jira/browse/TOMEE-1723

Until hazelcast fixes the lifecycle of its instance through a CDI extension it is likely the cleanest you can do.

Note: this is also doable using tomee internal server API to start the instance earlier but not needed for most cases

Halothane

Oh, I did not know that the HazelcastServerCachingProvider will actually use CDI to get the HazelcastInstance. I will try that first thing tomorrow. Thanks for your quick reply. – Benjie

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Stamford

I've updated my testcase by adding an HazelCastInstanceManager. Sadly, that only leads to two Hazelcast instances being created, one of which does not stop correctly (presumably the JCache one). I will investigate if there is any possibility to have the HazelcastServerCachingProvider to use a HazelcastInstance that is managed by the HazelcastInstanceManager. – Benjie

you need to ensure both configuration match (see github.com/hazelcast/hazelcast/blob/master/hazelcast/src/main/…) otherwise you will create one instance and JCache impl another one – Halothane

Answer 2

Solution

rmannibucau's answer pointed me to the right direction.

I added a bean that @Observes @Destroyed(ApplicationScoped.class) and calls Caching.getCachingProvider().close(). This in turn shuts down the underlying Hazelcast instance.

This solution also avoids direct interaction with Hazelcast classes. The dependency can remain limited to the runtime scope.

I added a branch to the test case with this solution.

Benjie