Creating an Aurora Serverless Cluster from cloudformation?

Question

From Aurora Serverless's document, there are 3 ways to create an Aurora serverless DB cluster: AWS management console, CLI, and RDS API. (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/aurora-serverless.create.html)

Form my understanding, one would use EngineMode in the RDS API to create Aurora Serverless, but this property is not available in AWS::RDS::DBCluster yet (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html).

Would it be possible to create an Aurora Serverless Cluster from cloudformation? Any advice would be appreciated!

Orbicular

Answer 1

Thanks for Chris's update. As an example, here is my cloudFormation template for serverless aurora. We no longer need the DBInstance.

  RDSCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      MasterUsername: 
        Ref: DBUsername
      MasterUserPassword: 
        Ref: DBPassword
      DatabaseName: RANDOMNAME
      Engine: aurora
      EngineMode: serverless
      ScalingConfiguration:
        AutoPause: true
        MaxCapacity: 16
        MinCapacity: 2
        SecondsUntilAutoPause: 300
      DBSubnetGroupName:
        Ref: DBSubnetGroup

More complete example of all available options for RDS (including Aurora): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html

Orbicular

Thanks for the example, I feel like there are missing things there though, like the Subnet, you use a Ref but no example of how to configure it. Could you update it with a more thorough example if that's the case? Thanks! – Frasch

I'm also in the same boat of @Vadorequest, could you go further in the DBSubnetGroup, please? – Tabbie

Thanks for asking! My DBSubnetGroup is essentially the same as the example here: docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… – Orbicular

From what I can tell, creating subnets and VPCs from SAM/Cloudformation is complicated. Based on the fact that most people don't create them in SAM and that they are easily re-usable across stacks, I think most people either just use the default one, create a new one using the GUI. I went and found subnet IDs for my default VPC here: console.aws.amazon.com/vpc/home?region=us-east-1#subnets: – Algonquin

Answer 2

It is now possible to create an AWS::RDS::DBCluster with an EngineMode set to serverless. See more here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-enginemode

Burdine

Answer 3

Simple answer - No. Not till they make it available in CFN. As of 1-2 days ago, the EngineMode and ScalingConfiguration property are not yet available in the RDS API, as my API call threw this error. First they will make available the APIs/cli. Once that works, create a CFN Custom Resource to invoke the RDS API from a lambda. It might be a while before it is made directly available in CFN.

2018-08-15T16:12:09.648Z f57erb2b-g3a5-11e8-8f64-81912181e535 { MultipleValidationErrors: There were 2 validation errors: * UnexpectedParameter: Unexpected key 'EngineMode' found in params * UnexpectedParameter: Unexpected key 'ScalingConfiguration' found in params

And I know role/permission is not a problem as I could launch normal Aurora cluster from the same.

PS: RDS APIs now work for serverless

Edit: Some time in Oct 18, EngineMode added to CFN, so now this is possible -> https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-enginemode

Manic

for others looking at this answer you can see if EngineMode has been added to CFM at this link: docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… As of Aug 2018 it's still not available – Oswald

Answer 4

I found an article that indicated that the Go SDK has been updated with the EngineMode parameter, and when I looked through the AWS SDK changelog, I found some RDS updates in the most recent versions. I've had to manually update my AWS CLI to the latest release to get my shell scripts working with that option.

https://github.com/aws/aws-cli/releases

https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst

https://github.com/terraform-providers/terraform-provider-aws/issues/5503

No news on the CloudFormation side though.

Sweetie

I just tried out the Go SDK yesterday to create a serverless cluster and it worked. I’m new to RDS, so I have checked to see if everything is set up correctly, but on the surface, it seems to work. – Amarillas

Answer 5

For Aurora Postgres Serverless, my full, working DBCluster resource is:

  RDSCluster:
    Type: AWS::RDS::DBCluster
    Properties:
      DBClusterIdentifier: !Ref DBClusterName
      MasterUsername: some-name
      MasterUserPassword: some-password
      DatabaseName: some-db-name
      Engine: aurora-postgresql
      EngineMode: serverless
      EngineVersion: '10' # this currently provisions '10.serverless_14'
      EnableHttpEndpoint: true # for HTTP API endpoint
      ScalingConfiguration:
        AutoPause: true
        MaxCapacity: 2
        MinCapacity: 2 # min 2 currently
        SecondsUntilAutoPause: 900 # 15 min
      DBSubnetGroupName:
        Ref: DBSubnetGroup

Epa

How here do you specify the endpoint itself? I am attempting to set this up but unable to link what you have above with a workable endpoint that I can call from my lambdas. – Cheng

To connect lambda to Aurora you can either 1) use the data API as you suggest or 2) for lambdas as I tend to put most services in the same VPC as my RDS then can still just add an SG to lambda allowing traffic out (eg over port 5432) and another SG to allow traffic in and out of RDS (on same port eg 5432) & it will connect as normal RDS. But to answer your question you can see the http endpoint in the AWS RDS console or just add it as an Output at the end of your SAM template.yaml - in fact here is a full (AWS authorized) example: serverlessland.com/patterns/lambda-aurora – Epa

Thanks, great resource you shared @Leigh ! – Cheng

You’re welcome! Yes it’s an awesome site, the patterns/ templates and videos are brilliant, really speeds up my dev any micro-service I build always worth a check if there’s a template already – Epa