Disable redis when many timeouts using spring boot

Question

I have an application deployed in AWS EC2, some time (rare), I can't connect and execute any command in redis, I am investigating the root cause of this problem.

I am using Spring boot + Redis (Elasticcache).

I am using a wrapper to catch any exception to continue the request process.

My wrapper:

class RedisCacheWrapper implements Cache {

private final Cache delegate;

public RedisCacheWrapper(Cache redisCache) {
    Assert.notNull(redisCache, "delegate cache must not be null");
    this.delegate = redisCache;
}

@Override
public String getName() {
    try {
        return delegate.getName();
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public Object getNativeCache() {
    try {
        return delegate.getNativeCache();
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public ValueWrapper get(Object key) {
    try {
        return delegate.get(key);
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public <T> T get(Object o, Class<T> type) {
    try {
        return delegate.get(o, type);
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public <T> T get(Object o, Callable<T> callable) {
    try {
        return delegate.get(o, callable);
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public void put(Object key, Object value) {
    try {
        delegate.put(key, value);
    } catch (Exception e) {
        handleException(e);
    }
}

@Override
public ValueWrapper putIfAbsent(Object o, Object o1) {
    try {
        return delegate.putIfAbsent(o, o1);
    } catch (Exception e) {
        return handleException(e);
    }
}

@Override
public void evict(Object o) {
    try {
        delegate.evict(o);
    } catch (Exception e) {
        handleException(e);
    }
}

@Override
public void clear() {
    try {
        delegate.clear();
    } catch (Exception e) {
        handleException(e);
    }
}

private <T> T handleException(Exception e) {
    log.error("handleException", e);
    return null;
}}

In my redis configuration I set the timeout to 1s. So, when connect/command is not perform after 1s, the redis throws an Exception like this:

Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out

My doubt: There is a good way to disable cache temporary (without any deploy), while redis is not good? For example: Using a circuit break?

I am think do it:

    @Cacheable()
    myMethodCached(){
       myRealMethod();
    }

    myRealMethod(){}

Put "myMethodCached" in HystrixCommand, if timeout is throwed, then fallback method is performed without using redis.

The problem with this approach is that I need to create a "fallback" for all methods that use cache, I would like to "disable" globally (all cache will be skipped).

Is there a good solution to "disable" redis for a period?

Religiose

Answer 1

If you are using Spring Data Redis, you can leverage Spring's support for handling these temporary outages and exceptions via a custom exception handler.

Code:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Recommend to set the timeout lower than the default (60000):

spring.cache.type=redis
spring.redis.timeout=100

Then create a custom error handler in the Spring context:

import lombok.extern.slf4j.Slf4j;
import org.springframework.cache.Cache;
import org.springframework.cache.annotation.CachingConfigurerSupport;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cache.interceptor.CacheErrorHandler;
import org.springframework.context.annotation.Configuration;

@Slf4j
@EnableCaching
@Configuration
public class CacheConfiguration extends CachingConfigurerSupport {

    @Override
    public CacheErrorHandler errorHandler() {
        return new CacheErrorHandler() {
            @Override
            public void handleCacheGetError(RuntimeException exception, Cache cache, Object key) {
                log.info("Failure getting from cache: " + cache.getName() + ", exception: " + exception.toString());
            }

            @Override
            public void handleCachePutError(RuntimeException exception, Cache cache, Object key, Object value) {
                log.info("Failure putting into cache: " + cache.getName() + ", exception: " + exception.toString());
            }

            @Override
            public void handleCacheEvictError(RuntimeException exception, Cache cache, Object key) {
                log.info("Failure evicting from cache: " + cache.getName() + ", exception: " + exception.toString());
            }

            @Override
            public void handleCacheClearError(RuntimeException exception, Cache cache) {
                log.info("Failure clearing cache: " + cache.getName() + ", exception: " + exception.toString());
            }
        };
    }

}

Spring should detect the failure after 100 milliseconds and fallback to retrieve the data retrieved via @Cacheable annotated methods normally as if there were a cache-miss. And whenever cache is restored Spring will start pulling from cache again.

Mccraw

worked for me using spring boot 2.6.4 – Defloration

Tested on [email protected] and it works well but only if the Redis instance is still available. If you can't connect to it, Lettuce client will complain and calls will timeout (with a io.lettuce.core.RedisCommandTimeoutException exception being thrown). The error handler is never reached in this case, which is a shame because it means you have to have another fallback in case the instance is completely unavailable. – Jessicajessie

My bad, the problem I was facing was because I was also having Spring sessions (spring-session-data-redis) stored in Redis instance. – Jessicajessie

How about in this situation? I mean, how to handle redis timeout query when session is stored using redis? – Enactment