Spring Data Redis JedisConnectionException: Unexpected end of stream

Question

Redis 3.0.5
Spring Data Redis 1.3.6
jedis 2.6.3
- Our is web application which receives data from redis over pub/sub.
- Also performs read/write of data on redis as key/value pairs.
- read/write happens on listener thread, independent monitoring thread and http request threads.
- We have used a same connection factory for Listener and redis template
- Our redis server has "timeout=30" configured

<bean id="jedisConnectionFactory"
    class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
    <property name="hostName" value="nnnn"></property>
    <property name="port" value="nnnn"></property>
    <property name="password" value="****"></property>
</bean>
<bean id="redisTemplate" class="org.springframework.data.redis.core.StringRedisTemplate">
    <property name="connectionFactory" ref="jedisConnectionFactory" />
</bean>
<bean id="redisContainer"
    class="org.springframework.data.redis.listener.RedisMessageListenerContainer">
    <property name="connectionFactory" ref="jedisConnectionFactory" />
    <property name="messageListeners">
        <map>
            <entry key-ref="messageListener">
                <bean class="org.springframework.data.redis.listener.ChannelTopic">
                    <constructor-arg value="topic_name" />
                </bean>
            </entry>
        </map>
    </property>
    <property name="taskExecutor" ref="redisTaskExecutor" />
    <property name="subscriptionExecutor" ref="redisSubTaskExecutor" />
</bean>
<bean id="redisTaskExecutor"
    class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="threadNamePrefix" value="RedisListenerThread"></property>
    <property name="corePoolSize" value="1" />
    <property name="maxPoolSize" value="1" />
</bean>
<bean id="redisSubTaskExecutor"
    class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="threadNamePrefix" value="RedisSubscribeThread"></property>
    <property name="corePoolSize" value="1" />
    <property name="maxPoolSize" value="1" />
</bean>
<bean id="messageListener"
    class="org.springframework.data.redis.listener.adapter.MessageListenerAdapter">
    <constructor-arg index="0">
        <bean class="my.data.Receiver" />
    </constructor-arg>
    <constructor-arg index="1"><value>receive</value></constructor-arg>
</bean>

occasionally we face below problem on production during data read.

org.springframework.data.redis.RedisConnectionFailureException: Unexpected end of stream.; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream. at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47) at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36) at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37) at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37) at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:182) at org.springframework.data.redis.connection.jedis.JedisConnection.get(JedisConnection.java:1087) at org.springframework.data.redis.connection.DefaultStringRedisConnection.get(DefaultStringRedisConnection.java:276) at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:46) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84) at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:43)

I have read other threads talking of using single thread for read/write. But in our case it is difficult to use single thread. Also as per RedisTemplate documentation, it is thread safe. The problem is occasional and we are unable to reproduce in any of dev/testing/uat environment. Thus unable to find exact cause of the same. What have we done wrong?

Oraleeoralia

Answer 1

We have been able to reproduce the issue and the cause is "timeout=30" setting in Redis.

Scenario

1. Connection is idle for 30seconds and Redis kills the same.
2. Before "Redis connection factory" in application detects the broken connection, it gets allocation for read or write request
3. Code tries to use this connection but as it is broken, it is unable to send command for read/write. Thus we get "JedisConnectionException: Unexpected end of stream" exception

Solution

1. set Redis timeout to Zero
2. Using custom JedisPoolConfig set the minEvictableIdleTimeMillis to desired value. This will ensure idle connections are released from Jedis connection pool

Oraleeoralia

Answer 2

For those guys, who runs redis in docker: do not forget to bind it to 0.0.0.0 instead of 127.0.0.1

Nichollenicholls

Answer 3

I'm posting this answer just for the purpose of knowledge as this is one of the possible scenario

Since you are using Pub Sub mechanism, one of the possible reason for this error is that input/output buffer of the application is more than the configured limit of the redis application

Redis has the hard limit of the 32mb and soft limit of 8mb for 60 seconds as per the redis documentation. So by changing the configuration we can get large throughput of data in redis pub sub mechanism.

You just have to update the configuraiton value of the below key

client-output-buffer-limit pubsub

to a larger value.

Spilt