How to disable default shutdown hook of Hazelcast in spring boot

Question

I am developing web socket application using Hazelcast to share the status of the online users. Everything works fine except one thing that is when one of the application instance goes down or restarts, all user connected to that instance get disconnected and afterConnectionClosed of MessagingHandler that extends BinaryWebSocketHandler. In afterConnectionClosed, the status of users which are connected to current node will be updated and these statuses are in Hazelcast. So when it attempts removal of the status from the Hazelcast, it gives the following error:

com.hazelcast.core.HazelcastInstanceNotActiveException: State: SHUT_DOWN Operation: class com.hazelcast.map.impl.operation.RemoveOperation
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.engineActive(Invocation.java:490)
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:523)
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:513)
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:207)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:60)
        at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:423)
        at com.hazelcast.map.impl.proxy.MapProxySupport.removeInternal(MapProxySupport.java:563)
        at com.hazelcast.map.impl.proxy.MapProxyImpl.remove(MapProxyImpl.java:207)
        at com.nisheeth.spring.MessageHandler.afterConnectionClosed(MessageHandler.java:57)
        at org.springframework.web.socket.handler.WebSocketHandlerDecorator.afterConnectionClosed(WebSocketHandlerDecorator.java:85)
        at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.afterConnectionClosed(LoggingWebSocketHandlerDecorator.java:72)
        at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.afterConnectionClosed(ExceptionWebSocketHandlerDecorator.java:78)
        at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.onClose(StandardWebSocketHandlerAdapter.java:141)
        at org.apache.tomcat.websocket.WsSession.fireEndpointOnClose(WsSession.java:535)
        at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:481)
        at org.apache.tomcat.websocket.WsSession.close(WsSession.java:445)
        at org.apache.tomcat.websocket.WsWebSocketContainer.destroy(WsWebSocketContainer.java:960)
        at org.apache.tomcat.websocket.server.WsContextListener.contextDestroyed(WsContextListener.java:48)
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4792)
        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5429)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1435)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
        at ------ submitted from ------.(Unknown Source)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:127)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:147)
        at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:424)
        at com.hazelcast.map.impl.proxy.MapProxySupport.removeInternal(MapProxySupport.java:563)
        at com.hazelcast.map.impl.proxy.MapProxyImpl.remove(MapProxyImpl.java:207)
        at com.nisheeth.spring.MessageHandler.afterConnectionClosed(MessageHandler.java:57)
        at org.springframework.web.socket.handler.WebSocketHandlerDecorator.afterConnectionClosed(WebSocketHandlerDecorator.java:85)
        at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.afterConnectionClosed(LoggingWebSocketHandlerDecorator.java:72)
        at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.afterConnectionClosed(ExceptionWebSocketHandlerDecorator.java:78)
        at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.onClose(StandardWebSocketHandlerAdapter.java:141)
        at org.apache.tomcat.websocket.WsSession.fireEndpointOnClose(WsSession.java:535)
        at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:481)
        at org.apache.tomcat.websocket.WsSession.close(WsSession.java:445)
        at org.apache.tomcat.websocket.WsWebSocketContainer.destroy(WsWebSocketContainer.java:960)
        at org.apache.tomcat.websocket.server.WsContextListener.contextDestroyed(WsContextListener.java:48)
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4792)
        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5429)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1435)
        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

I am disabling default shutdown using following properties:

config.setProperty(GroupProperty.SHUTDOWNHOOK_ENABLED.getName(), "false");
config.setProperty(GroupProperty.SHUTDOWNHOOK_POLICY.getName(), "GRACEFUL");

But still it is still shutdown before the users were disconnecting. Is there a way to configure hazelcast in a way that hazelcast shutdown in the end of the application?

Malca

Answer 1

I have been trying different methods to handle graceful shutdown of spring websocket connection which also modifies the hazelcast map of online users, but none worked. I finally used SmartLifecycle interface. Documentation of this class can be found here.

Here is the code that I used for proper shutdown. And yes, it is called before Hazelcast shutdown:

@Component
public class AppLifecycle implements SmartLifecycle {

    private Logger log = LoggerFactory.getLogger(AppLifecycle.class);

    @Autowired
    private SampleService sampleService;

    @Override

    public boolean isAutoStartup() {
        log.debug("=========================auto startup=========================");
        return true;
    }

    @Override
    public void stop(Runnable runnable) {
        sampleService.getSessionMap().forEach((key, session) -> {
            try {
                session.close(CloseStatus.SERVICE_RESTARTED);
                log.debug("disconnecting : {}", key);
            } catch (IOException e) {
                log.error(e.getMessage(), e);
            }
        });
        log.debug("=========================stop runnable=========================");
        new Thread(runnable).start();
    }

    @Override
    public void start() {
        log.debug("=========================start=========================");
    }

    @Override
    public void stop() {
        log.debug("=========================stop=========================");
    }

    @Override
    public boolean isRunning() {
        return true;
    }

    @Override
    public int getPhase() {
        return Integer.MAX_VALUE;
    }
}

Malca

Answer 2

@nisheeth-shah, It's more like a Spring related issue, since it's Spring that decides the shutdown order of the beans.

What you can do, you can annotate your MessageHandler bean with @DependsOn annotation and give name of the HazelcastInstance bean, like this @DependsOn("hazelcastInstance"). Please see this link for the explanation. Basically, Spring will start HazelcastInstance before creating MessageHandler and won't shut down HazelcastInstance before MessageHandler bean destroyed.

Also, don't disable Hazelcast shutdown hook.

Giesecke

link is not working as expected – Farland

Answer 3

You can use hazelcastInstance.shutdown() in @PreDestroy ie.

@PreDestroy
public void shutDownHazelcast(){
        hazelcastInstance.shutdown()
}

This will shut down you the hazelcast instance gracefully

Courland

It isn't working at all. I have tried this option. The hazelcast shuts down before user gets disconnected gracefully. – Malca