Suddenly I'm getting "Hazelcast instance is not active!"

Question

My application is running for months and working very well. Then suddenly I get the following error:

com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:220)
    at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:207)
    at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:69)
    at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:67)
    at com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:47)
    at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:101)
    at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:285)
    at com.hazelcast.instance.HazelcastInstanceImpl.getLock(HazelcastInstanceImpl.java:183)
    at com.hazelcast.instance.HazelcastInstanceProxy.getLock(HazelcastInstanceProxy.java:77)
    at br.com.xyz.lock.hazelcast.HazelcastLockManager.lock(HazelcastLockManager.java:37)
    at br.com.xyz.lock.hazelcast.LockManagerFacade.lock(LockManagerFacade.java:24)
    at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processamentoLock(NProcessadorMensagemRecebida.java:85)
    at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processaArquivo(NProcessadorMensagemRecebida.java:74)
    at br.com.xyz.recebe.processador.ProcessadorBase.processaArquivo(ProcessadorBase.java:75)
    at br.com.xyz.recebe.processador.ProcessadorXml.processaArquivo(ProcessadorXml.java:16)
    at br.com.xyz.recebe.processador.ProcessadorFacade.processaArquivo(ProcessadorFacade.java:34)
    at br.com.xyz.recebe.mail.pdes.ProcessadorPDESMeRecebida.processar(ProcessadorPDESMeRecebida.java:77)
    at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.processar(ProcessadorDiretorioEntradaSaidaDaemon.java:575)
    at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.varrerDiretorioUsingStrategy(ProcessadorDiretorioEntradaSaidaDaemon.java:526)
    at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.run(ProcessadorDiretorioEntradaSaidaDaemon.java:458)
    at java.lang.Thread.run(Unknown Source)

I found some issues on Google that says it shutdown because other errors. But in my case there isn't any.

It shutdown without reason.

Has anyone seen this before?

Piggy

Are you sure that Hazelcast-server is running? – Rattler

Write test program with new client and try to connect to your Hazelcast node. Check it. – Rattler

If test will be ok, then problem maybe in your Hazelcast client... maybe it was shutdown with OutOfMemory or some other problems. – Rattler

It working for a long time. The instance was up and the node is running. But I dont get why it shutdown. I get others errors in my log, but nothing unusual. – Piggy

Maybe this link will be useful: grepcode.com/search/… – Rattler

@DmitryTsechoev but how I recover from that error? How I can check if the node is shutted down? – Piggy

I assumed that in the source code, you can track the location where the exception is thrown. And on this basis, you can gradually try to understand the reason. Rather, the problem on the client side. You will not believe, but just got a message from the head of the exact same error in our production area! ) – Rattler

Not so long ago we investigated a similar problem, and if I'm not mistaken, the reason could be a short outage is Hazelcast node, after which the client falls here with this error. Run locally Hazelcast node, connect by client, turn off Hazelcast node and look at the log of the client. – Rattler

@DmitryTsechoev I'm not using hazelcast-client. I just do lock and unlock in a value for synchronization: hazelcastInstance.getLock(value).lock(); and hazelcastInstance.getLock(value).unlock(); – Piggy

Then I can not help. Sorry. – Rattler

Answer 1

The problem occurs if a hazelcast member doesn't shutdown normally (terminate), When you stop HazelcastInstance which queue proxy is bound to; then any operation on that queue after instance stopped should throw HazelcastInstanceNotActiveException.

Yeung

But I dont get why it shutdown that way. I'm sure is not OutOfMemory because my logs dont shown. – Piggy

I currently using the version 3.1.5. I think it was solved after version 3. Do you know how to recover from that error? – Piggy

You are right! This happened only when application is finishing. Thats not a problem so far. – Piggy

Answer 2

You just need to wait for a few minutes and rerun your job.

Lello

Answer 3

in my case, such an error in the logs began to occur after restarting the application on one of the nodes, while the sh script worked correctly. and he showed about the correct closing of the application and also about the correct start. But in the end, somehow the application hung up when closing, the process was duplicated and there were 2 active applications on the same node. It helped me to kill 2 processes and start the application again. An error can also occur when: connectivity was disrupted, or something with resources and disabled the hazlecast, or it did not find the second node.

Weis