How to change AWS-RDS instance type without downtime?

Question

I'm using AWS-RDS(Aurora MySQL5.6) and It's a cluster, it has one writer instance and one reader instance. I find each instance would be down for nearly 10 minutes when I change its type, it's unacceptable, what should I do to avoid downtime?

Here are more details information:

• Change type from db.t2.small to db.t2.medium
• Engine version: 5.6.10a
• Freeable Memory: 300M
• DB Connections (Count): ~400
• CPU Utilization (Percent): ~20%

Bluefish

Answer 1

Instead of changing the type of the current writer instance, add a new writer with the desired instance type:

1. Add an Aurora reader with the desired new instance type (for example db.t2.medium) to the cluster. Be sure to assign the reader the best priority so that it will be promoted to writer during failover (see Aurora Documentation for more in-depth information).
2. Wait until the new instance is up and running.
3. Then failover the cluster so that the new reader gets promoted to a writer.
4. Finally delete the previous Aurora instance of the old instance type.

This way you will have nearly no downtime.

If you want to change the type of a reader instance:

1. Add an Aurora reader with the desired new instance type.
2. Wait until it's up and running.
3. Delete the previous reader instance.

Junkman

How about the writer instance? – Bluefish

For clarification, I added a distinction between writer and reader in the answer. – Junkman

@Bluefish did the answer help with your question? If yes it would be nice if you accepted the answer. – Junkman

Sorry, I haven’t tried it yet, I’ll do it tomorrow~ – Bluefish

Why adding of new reader is necessary? Adding operation is always recommended but what problem happens if I change type of existent reader directly? Writer is still accessible at this time. – Dorey

@Dorey if you have a cluster with a writer and a reader instance and change the instance type of the reader instance, this reader instance will be down for a period of time and requests to this instance will not be successful. So I would not recommend that if you need zero downtime. – Junkman

@Junkman 1. Isn't requests will be redirected to writer if reader is stopped? 2. Isn't requests usually comes to writer EVEN IF reader is running (because reader data every time little bit older (sometimes seconds are critical) than writer before it will be replicated.) [of course if you set DB endpoint to cluster URL in your app]? 3. Reader is meaningless at all if it don't failover when stopped. 4. Zero downtime is not possible with AWS cluster in any case: downtime appearing when you try failover previous writer to new reader (to change writer→reader) – Dorey

@Junkman PS Actually I have a question about cluster downtime, and looks like no one knows an answer, if you have some information about please share it. – Dorey

@Dorey 1) As far as I know, there is no such redirection. – Junkman

4) Yes, there is small downtime during failover due to reader/write restart. I'd suggest you look into RDS Proxy if you haven't already done so. It can reduce the downtime in your client applications due to slow DNS propagation. There is a good introduction article about it which shows the speedup from 24 to 3 seconds average failover recovery time when using RDS proxy. – Junkman

@Junkman Ok. I have checked info about cluster endpoints (I had to start from this). Reader database is not used at all if only cluster endpoint used. And reader may be stopped safely (of course writer must be online through stopping time). If reader endpoint used for loadbalancing and SELECT queries, new reader instance should be created to decrease service offline time (when reader stopped). – Dorey

Also note you can pick the instance to fail over to, so promoting a new instance to the writer doesn't have to rely on fine-tuning the priority tiers for all the instances. – Coorg

Answer 2

Actually this sounds like an ideal use case for Serverless v2. You would need to be running Aurora MySQL v3 (MySQL 8.0-compatible). But you can say "I want to run somewhere between 2 GB and 4 GB of RAM" (1-2 ACUs) and Aurora decides when the instance is busy enough or idle enough to scale up or down. Connections and transactions are preserved while the scaling happens, so no downtime. You can pick Serverless v2 for the writer, the reader, or both. And you can switch to Serverless v2, or back to predefined instance classes if desired, without having to re-create the cluster.

Instructions: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.html

Coorg

Answer 3

I am assuming you are using cloud formation or terraform to manage db cluster and instances. I think, adding a "dependsOn" in cloud formation might help you in doing a sequential update.

e.g. instance 1(reader) depends on instance 2 (writer)

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html

Godred