How to do a redis FLUSHALL without initiating a sentinel failover?

Question

We have a redis configuration with two redis servers. We also have 3 sentinels to monitor the two instances and initiate a fail over when needed.

We currently have a process where we periodically have to do a FLUSHALL on the redis server. This is a blocking operation that takes longer than the time we have allotted for the sentinels to timeout. In other words, we have our sentinel configuration with:

sentinel down-after-milliseconds OurMasterName 5000

and doing a redis-cli FLUSHALL on the server takes > 5000 milliseconds, so the sentinels initiate a fail over.

We acknowledge that doing a FLUSHALL isn't great and we also know that we could increase the down-after-milliseconds to but for the purposes of this question assume that neither of these are options.

The question is: how can we do a FLUSHALL (or equivalent operation) WITHOUT having our sentinels initiate a fail over due to the FLUSHALL blocking for greater than 5000 milliseconds? Has anyone encountered and solved this problem?

Genarogendarme

if you're on some cloud platform, you could just create a new instance: either have machine images ready or with some devops tools – Rigatoni

@LiviuCostea I think this is probably the right option. If you can reference something that describes in a little more detail how this could work I would be happy to accept your answer. – Genarogendarme

If you are using something like AWS or Azure than you have API for creating a new Redis cluster. Start it, load it with data and once ready just modify the DNS, again with API call -so all these can be handled by some part of your application. But on premises things can get more complex because it will require some automation with ansible/chef/puppet. – Rigatoni

@LiviuCostea True -- still a legit answer. If you put this in answer form I can accept it. – Genarogendarme

Answer 1

You could just create new instances: if you are using something like AWS or Azure than you have API for creating a new Redis cluster. Start it, load it with data and once ready just modify the DNS, again with API call -so all these can be handled by some part of your application. But on premises things can get more complex because it will require some automation with ansible/chef/puppet.

Rigatoni

Answer 2

The next best option you currently have to is to delete keys in batches to reduce the amout of work to at once. You can build a list, assuming you don't have one, using scan Then delete in whatever batch size works for you.

Edit: as you are not interested in keeping data, disable persistence, delete the RDB file, then just restart the instance. This way you do t have to update sentinel like you would if you take the provision new hosts.

Out of curiosity, if you're just going to be flushing all the time and don't care about the data as you'll be wiping it, why bother with sentinel?

Gloze

I think the problem here is that it would not guarantee consistency. In this case we are probably better off just increasing the timeout to > 5000 ms (which we actually did -- and temporarily solves our problem -- but not really an acceptable solution). – Genarogendarme