Hazelcast prevents the JVM from terminating

Question

We use Hazelcast 2.6.2 in a legacy Java clustered application. When the application is stopped the JVM does not terminate any more. It seems that it is caused by Hazelcast threads not being flagged daemon. I did not find a way way through the Hazelcast API to flag them daemon.

Are there recommended solutions to prevent Hazelcast from preventing the JVM to terminate?

Regards

Sloth

"when the application is stopped": How? Do you simply end all non-daemon threads or do you actually use System.exit()? The latter should always work, no matter if any daemon threads are running or not. – Barnard

Are you calling hazelcast.shutdownAll();? – Nijinsky

@JoachimSauer: all non-daemon threads simply end, otherwise a simple System.exit() would be enough to terminate the JVM like you said – Kelleher

@Gray: unfortunatelly I do not know when the application ends, there is no callback for me to execute hazelcast.shutdownAll() or even a System.exit() – Kelleher

I've added an answer that shows how to use the shutdown hook. – Nijinsky

Answer 1

Looking at the Hazelcast Javadocs, I see that there is a shutdownAll(); method. To quote the javadocs:

Shuts down all running Hazelcast Instances on this JVM, including the default one if it is running. It doesn't shutdown all members of the cluster but just the ones running on this JVM.

If you aren't shutting it down I suspect there are non-daemon threads there that aren't being terminated that will keep the JVM from closing.

Nijinsky

I think this won't work. The shutdown hook will never get executed since the JVM will not terminate. :'( – Kelleher

I think hazelcast.shutdownAll() would allow the JVM to terminate (when the application threads end of course) and so would trigger the shutdown hook. Not the other way around. – Kelleher

Duh. You are right @Tom. I'm afraid you are going to have to somehow trigger a call to shutdownAll() when the application is quitting. – Nijinsky

Correct approach. You need to detect somehow that your application is shutting down. If you're using a framework, look for appropriate shutdown hooks and call Hazelcast shutdownAll(). – Cabinetmaker

Answer 2

In Tomcat, I added a lifecycle listener via server.xml.

<Listener className="com.mycompany.StartupHandler" DEBUG="false"/>

My StartupHandler.class went in a jar in $TOMCAT_HOME/lib/, and it contains this snippet to trigger Hazelcast shutdown when Tomcat shutdown is detected:

public void lifecycleEvent(LifecycleEvent lifecycleEvent) {
    String eventType = lifecycleEvent.getType();
    if (eventType.equals(Lifecycle.BEFORE_STOP_EVENT)) {
        com.hazelcast.core.Hazelcast.shutdownAll();
    }
}

Other web servers should have similar shutdown hooks where you can invoke Hazelcast.shutdownAll() automatically.

See https://github.com/hazelcast/hazelcast/issues/718 for my original Hazelcast thread on the same topic. Thanks for the hint here to call Hazelcast.shutdownAll(). That idea combined with my Tomcat shutdown hook should be sufficient to resolve this problem for me, and hopefully you find it useful too.

Shopper

Thank you for that interesting idea. Unfortunately it is not applicable here since we do not use Tomcat. We need a portable solution. – Kelleher

Answer 3

Here is a Tomcat independent way to detect JVM shutdown. It uses a non-daemon thread to poll a daemon thread to detect when the JVM is shutting down (the JVM shuts down daemon threads automatically). Poll the static IS_SHUTDOWN flag in your code, or inline the call to Hazelcast.shutdownAll() here. Be careful of a race condition, because if you try to shutdown Hazelcast before it starts then it throws an exception in your log. The exception does not break anything, but it looks ugly.

/**
 * Non-Daemon thread monitors a sacrificial, low-priority, daemon thread
 * to detect when JVM is shutting down, so shutdown hooks can be invoked.
 * @author justin.cranford
 */
public class ThreadUtil {
    public static boolean IS_SHUTDOWN = false;  // threads can poll this flag, or inline below where flag is set
    private static final Thread DAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                try {
                    Thread.sleep(Long.MAX_VALUE);   // sleep forever
                } catch(Exception e) {}
            }
        }
    };
    private static final Thread NONDAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                if (!DAEMON_THREAD.isAlive()) { // poll forever, until daemon thread dies
                    ThreadUtil.IS_SHUTDOWN = true;
                    return;
                }
                try {
                    Thread.sleep(1000); // poll every 1000msec = 1sec
                } catch(Exception e) {}
            }
        }
    };
    static {
        DAEMON_THREAD.setName("ShutdownMonitorNonDaemonThread");
        DAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        DAEMON_THREAD.setDaemon(true);
        DAEMON_THREAD.start();
        try {
            Thread.sleep(3000); // wait 3000msec = 3sec before monitoring
        } catch(Exception e) {}
        NONDAEMON_THREAD.setName("ShutdownMonitorDaemonThread");
        NONDAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        NONDAEMON_THREAD.setDaemon(false);
        NONDAEMON_THREAD.start();
    }
}

Shopper

I think this won't work either. The non-daemon thread you use will prevent the JVM from terminating (according to geekexplains.blogspot.fr/2008/06/… ), so the daemon thread will never stop, so we won't detect the "shutdown". – Kelleher

Answer 4

As everyone said, hazelcastInstance.shutdownAll() is the solution.

But, I would to get a new feature in Hazelcast - provide a demon client as well. There are many use cases where cache instance needs to be shutdown as soon as application ends.

Bohun