Automatically reconnect Storm Topology to Redis Cluster on Redis restart

Question

I have created a Storm topology which connects to Redis Cluster using Jedis library. Storm component always expects that Redis is up and running and only then it connects to Redis and subscribes the events.Currently we use pub-sub strategy of Redis.

Below is the code sample that explains my Jedis Connectivity inside Storm to for Redis.

try {
    jedis.psubscribe(listener, pattern);
} catch(Exception ex) {
    //catch statement here.
} finally {
    pool.returnResource(jedis);
}

....

pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port

ListenerThread listener = new ListenerThread(queue, pool, pattern);
listener.start();

EXPECTED BEHAVIOUR

Once Redis dies and comes back online, Storm is expected to identify the status of Redis. It must not need a restart in case when Redis die and come online.

ACTUAL BEHAVIOUR

Once Redis restarts due to any reason, I always have to restart the Storm topology as well and only then it starts listening back to Redis.

QUESTION

How can I make Storm listen and reconnect to Redis again after Redis is restarted? any guidance would be appreciated, viz. docs, forum answer.

Matchbook

Answer 1

1. Catch the exception for the connection lost error and set the pool to null
2. (Assume that you doing this in Spout) Use an if-else statement to check if pool is null then create a new instance of JedisPool() assigning to the pool like in your code:

pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port

3. If pool not null (means connected) then continue your work

Abb

Answer 2

This is a common issue with apache-storm where connection thread is alivein stale condition, although the source from where you are consuming is down/restarted. Ideally it should retry to create new connection thread instead reusing the existing one. Hence the Idea is to have it automated it by by detecting the Exception (e.g. JMSConnectionError in case of JMS).

refer this Failover Consumer Example which will give you brief idea what to do in such cases.(P.S this is JMS which would be JMS in redis your case.)

The Steps would be something like this.

1. Catch Exception in case of ERROR or connection lost.
2. Init connection (if not voluntarily closed by program) from catch.
3. If Exception got to step 1.

Corey