Can MWAA requirements.txt be automatically set to the latest version?

Question

We have an automated upload system for our DAG's to MWAA.

Is there a way to have the requirements.txt in the S3 bucket be automatically set to its latest version?

I cannot find an option for this in the AWS console.

Identic

Answer 1

It's in the setup of MWAA. Edit MWAA instance, in the requirements.txt field, set the version to the latest.

Also,

MWAA uses requirements file to create the container image. So when you upload requirement file and specify it in the edit options and save, images are created(it takes couple of minutes to do it). Ideally image should only be updated when you have new python libraries to add. if you still want to automate, try to use lambda S3 trigger and use mwaa cli to trigger the update-environment command.

Imbecility

It does allow me to select the most recent version in the bucket, but the question is asking if there is a method to automatically set the requirements.txt to the latest version – Identic

What do you mean automatically, When you upload a new file, it is automatically tagged as the latest version. – Imbecility

When I upload a new requirements.txt, I want that to be set as the requirements.txt used by MWAA without me having to go into the AWS console and set it myself manually – Identic

MWAA uses requirement file to create the container image. So when you upload requirement file and specify it in the edit options and save, images are created(it takes couple of minutes to do it). Ideally image should only be updated when you have new python libraries to add. if you still want to automate, try to use lambda S3 trigger and use mwaa cli to trigger the update-environment command. – Imbecility

Thanks! If you update your answer to include your previous comment, I will mark it accepted :) – Identic

Answer 2

I did it using github actions jobs. This job also updates the plugins.zip version to the latest, if you find it useful:

update_mwaa_environment:
  runs-on: ubuntu-latest

  steps:
    - name: Checkout code
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.MWAA_DAG_AUTOMATION_ACCESS_KEY }}
        aws-secret-access-key: ${{ secrets.MWAA_DAG_AUTOMATION_SECRET_KEY }}
        aws-region: us-east-2

    - name: Get S3 Object Version for Requirements
      id: req_version
      run: |
        version=$(aws s3api head-object --bucket ${{ secrets.S3_BUCKET }} --key requirements.txt --query VersionId --output text)
        echo "req_version=$version" >> $GITHUB_OUTPUT

    - name: Get S3 Object Version for Plugins
      id: plugins_version
      run: |
        version=$(aws s3api head-object --bucket ${{ secrets.S3_BUCKET }} --key plugins.zip --query VersionId --output text)
        echo "plugins_version=$version" >> $GITHUB_OUTPUT

    - name: Update MWAA Environment
      run: |
        aws mwaa update-environment --name ${{ vars.MWAA_ENV_NAME }} --source-bucket-arn "arn:aws:s3:::${{ secrets.S3_BUCKET }}" --plugins-s3-object-version ${{ steps.plugins_version.outputs.plugins_version }} --requirements-s3-object-version ${{ steps.req_version.outputs.req_version }}

The MWAA_DAG_AUTOMATION_ACCESS_KEY and MWAA_DAG_AUTOMATION_SECRET_KEY must belong to a user that have the permissions to update the desired MWAA environment AND to describe the environment subnets:

  {
    Sid    = "MWAAUpdateEnvironment"
    Effect = "Allow"
    Action = [
      "airflow:UpdateEnvironment"
    ]
    Resource = [
      "arn:aws:airflow:${var.mwaa_aws_region}:${var.aws_account_id}:environment/${var.customer_id}"
    ]
  },
  {
    Sid    = "MWAADescribeSubnets"
    Effect = "Allow"
    Action = [
      "ec2:DescribeSubnets"
    ]
    Resource = [
      "*"
    ]
  }

In my case, I just gave it permission to describe all subnets, but it's not a best security practice, even in a controlled environment.

Codd