Run DB Migration in AWS Lambda using Alembic framework

Question

just a follow-up question to this one

I have the app deployed to AWS Lambda, which works with RDS Database PostgreSQL, Lambda is called by API Gateway for processing external requests like

GET /api/endpoint

Current DB migrate / App update flow:

0 - previous App code deployed in Lambda, it has ORM in it, working with DB, Lambda called by API Gateway

1 - in CI/CD DB Migration Lambda is executed during deploy, it updates Database Schema

2 - in CI/CD updated version of App is deployed, compatible with updated Database Schema.

The problem is between steps 1 and 2 code in deployed Lambdas sometimes it not compatible with updated Database Schema, causing errors on DB queries.

Any suggestion how to overcome this downtime window?

Toluidine

Answer 1

Im working with a similar aproach:

1. a lambda which excecutes business logic
2. an sqlalchemy layer (+ models definitions)
3. another lambda which executes the database migration each time the pipeline detects a new change on that layer folder

so: for deploying the first lambda im using SAM, it contains a parameter that indicates the layer version (lets call this file template.yaml):

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
    LayerVersion:
        Type:String

Then i reference it from the business logic lambda:

  LambdaExample:
    Type: AWS::Serverless::Function
    Properties:
      Role: !GetAtt LambdaRole.Arn
      FunctionName: lambdaexample
      CodeUri: ./functions/lambda_example
      Handler: lambda_function.lambda_handler
      Runtime: python3.9
      Timeout: 15
      Environment:
        Variables:
          DB_SECRET_NAME: !Ref SecretNameExample
      VpcConfig:
        SubnetIds: 
          - !Ref LambdaSubnetId1
          - !Ref LambdaSubnetId2
        SecurityGroupIds: 
          - !Ref LambdaSecurityGroupId
      Layers:
        - !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:layer:database-layer:${LayerVersion}

for the layer and the migration lambda i have another sam template, why? because i want to update the database model before deploying any lambda that use it and avoid failures that may happen if i reference the latest layer version without migrating it

so: this is an example of layer-template.yaml (ommiting role creations and so on)

DatabaseLayer:
    Type: AWS::Serverless::LayerVersion
    Name: 'database-layer'
    Properties:
      LayerName: 'database-layer'
      ContentUri: ./database-example
      CompatibleRuntimes:
        - python3.9
    Metadata:
      BuildMethod: python3.9

MigrateDatabaseFunction:
    Type: AWS::Serverless::Function
    Properties:
      Role: !GetAtt MigrateDBRole.Arn
      FunctionName: automigrate-database
      CodeUri: ./automigrate-database
      Handler: lambda_function.lambda_handler
      Runtime: python3.9
      Timeout: 120
      MemorySize: 2048
      Layers:
        - !Ref DatabaseLayer
      Environment:
        Variables:
          SECRET_NAME: !Ref SecretNameExample
      VpcConfig:
        SubnetIds: 
          - !Ref LambdaSubnetId1
          - !Ref LambdaSubnetId2
        SecurityGroupIds: 
          - !Ref LambdaSecurityGroupId

What should do the ci/cd from now on? everytime it detects changes on folder database-example it should invoke MigrateDatabaseFunction:

• build your layer/layer-template.yaml and deploy it
• execute a git diff of your database models file, if it was changed:
• execute a aws lambda invoke --function-name automigrate-datbase --region {{your_region}} and save the results in a file, so you can check if it failed or not
• if it wasnt fail you can build and deploy your template.yaml with the new layer version on its parameters

Odontalgia