Is it possible to access the Airflow API in AWS MWAA?

Question

I have an AWS MWAA Airflow v2.0.2 instance running.

I would like to have access to the Airflow API. Is this something supported currently? Planned for future releases? There is no mention of this in the AWS docs.

It looks like it was possible to enable the experimental API in AWS MWAA Airflow v1.10.12, but when I try to add api.auth_backend = airflow.api.auth.backend.default in the AWS UI, i get this error message:

Some of the provided configurations belong to the blocklist and can not be applied: api.auth_backend

Newmown

Seems to be asked in #66345488 – Fingerstall

Yes, but this gives access to the Airflow CLI, not the Airflow API. Quite a big difference in features actually! – Newmown

There are two options: a) MWAA does not support Airflow API b) try using auth_backend = airflow.api.auth.backend.basic_auth because the default option can be deny_all as described in airflow.apache.org/docs/apache-airflow/stable/security/api.html (I don't remember when we changed the default) Also - consider migrating to 2.0 as 1.10.X version will reach EOL on 17th of June this year. – Fingerstall

I am already running v2.0.2. And as mentioned in the post, this api.auth_backend is blocklisted by AWS. – Newmown

If any of the answers met the question, please accept and close the topic. – Mackler

Answer 1

According to THIS SESSION of Airflow Summit 2021 (around 41:15, when attendee ask about API of Airflow 2.0 and security concern). It seems AWS block REST API for security reason at the moment.

Booth

Is this restriction still in place? – Waxplant

Answer 2

We were able to run some API calls on MWAA CLI as described in the official AWS MWAA User Guide.

Unfortunately, not all Airflow API commands are supported by the MWAA CLI, but the documentation is quite clear about that.

Aracelyaraceous

Answer 3

Here is how to run a DAG on an AWS MWAA instance using a Node.js Lambda function without changing the api.auth_backend setting:

const axios = require('axios');
const { MWAAClient, CreateCliTokenCommand } = require('@aws-sdk/client-mwaa');

const client = new MWAAClient({ region: 'us-east-2' });

const getAirflowCliToken = async (environmentName) => {
  const command = new CreateCliTokenCommand({ Name: environmentName });
  const { CliToken, WebServerHostname } = await client.send(command);

  return { token: CliToken, host: WebServerHostname };
};

const triggerDag = async (environmentName, dagName, payload) => {
  // Get an Airflow CLI token.
  const { token, host } = await getAirflowCliToken(environmentName);

  const url = `https://${host}/aws_mwaa/cli`;
  const formattedPayload = JSON.stringify(payload);
  const data = `dags trigger -c '${formattedPayload}' ${dagName}`;
  const options = {
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'text/plain'
    }
  };

  await axios.post(url, data, options);
};

const handler = async (event) => {
  const environmentName = 'example-environment'
  const dagName = 'example_dag';
  const payload = {
    task_payload: 'FOOBAR'
  };

  // Trigger Airflow DAG.
  await triggerDag(environmentName, dagName, payload);

  return {
    statusCode: 200,
    body: 'OK'
  };
};

module.exports.handler = handler;

Gesundheit

Chad Jhonson I did exactly the same as this request, but I keep getting 400 Airflow command parsing error response, is this something has to be configured in the dags ? although using python script actually is successfully running the dag using the same format of the command – Astrobiology

since the content type is text/plain, I managed to get this worked by axios.post(url, "<command>", options) – Astrobiology

Answer 4

AWS MWAA now allows you to use Airflow Rest API to manage Airflow resources. Check here

Freemanfreemartin