Redis MSET with TTL in Spring RedisTemplate

Question

I have got a Spring Boot application that needs to take millions of key value pairs and insert them into Redis.

Currently I'm using the multiSet method 1,000 key value pairs at a time.

@Autowired
private final StringRedisTemplate template;

...

Map<String, String> keyValuePairs = new HashMap<>();
template.opsForValue().multiSet(keyValuePairs);

However we also need to set a TTL for each pair. There doesn't seem to be a way to do this with multiSet. There is a way with set but this would have to be called millions of times so may not be efficient.

// For each key value pair
template.opsForValue().set(key, value, timeout, unit);

Does anyone know a way of doing this or using the set in a performant way?

Thanks

Analects

Answer 1

Pipelining should solve your problem; you create a pipeline, fill it with all the commands you want to perform, and then send those in bulk to redis. Using SET with EX you should be able to send 10,000 or more at a time.

Bo

FWIW, here's the link to the Spring Data Redis Pipelining feature documentation – Wanids

Answer 2

I was facing the same issue and finally solved it with Redis transaction. All the operations between operations.multi() and operations.exec() will be executed as one transaction, saves lot of network calls. You can implement rollback also with operations.discard()

List<Object> txResults = redisTemplate.execute(new SessionCallback<List<Object>>() {
                          public List<Object> execute(RedisOperations operations) throws DataAccessException {
                            operations.multi();
                            for (Map.Entry<String, Integer> entry : toUpsert.entrySet()) {
                                operations.opsForValue().set(entry.getKey(), entry.getValue(), expirySeconds, TimeUnit.SECONDS);
                            }
                            return operations.exec();
                          }
                        });

Spring Doc: Redis Transaction

Morin