How can I enable the API in AWS Managed Workflows for Apache Airflow?

Question

I'm testing the waters for running Apache Airflow on AWS through the Managed Workflows for Apache Airflow (MWAA). The version of Airflow that AWS have deployed and are managing for me is 1.10.12.

When I try to access the v1 REST API at /api/experimental/test I get back status code 403 Forbidden.

Is it possible to enable the experimental API in MWAA? How?

Lierne

Answer 1

I think MWAA provide a REST endpoint to use the CLI

https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli

It's quite confusing because you fisrt need to create a cli-token using the awscli to then hit the endpoint using that token. You will need a policy to allow your awscli to request that token. Lastly there isn't support for all the commands, just a bunch.

Anyway it's all explained on the user guide https://docs.aws.amazon.com/mwaa/latest/userguide/amazon-mwaa-user-guide.pdf

Manifesto

Thanks but I am still not able to hit the API endpoint. Could it be because I am on private webserver? @martin – Interrogate

You need to make the webserver public – Cyndi

Answer 2

By default, api.auth_backend configuration option is set to airflow.api.auth.backend.deny_all in MWAA environments. You need to override it to one of the authentication methods mentioned in the documentation as shown in the figure bellow:

Note: it is highly discouraged to use airflow.api.auth.backend.default as it'll leave your environment publicly accessible.

[2021/07/29] Edit: Based on this comment, AWS blocked access to the REST API.

Parkinson

Thank you @hedi-bejaoui. Perhaps I wasn't clear enough in my question. I am trying to get this done specifically in MWAA. The AIRFLOW__API__AUTH_BACKEND is not accessible for me to set in the MWAA settings page so I am asking whether there is another way for me to open up the API in MWAA. – Lierne

@Lierne I got your question since I was in a similar position too, probably my answer is the one who wasn't that clear. In order to override any config in the airflow.cfg file, you need to specify the section (the one between brackets) followed by the name of the config like this: api.auth_backend. – Parkinson

@Lierne The screenshot above is taken from the MWAA settings interface. – Parkinson

Is this really possible? In MWAA v2.0.2, when trying to enable this setting, I get Some of the provided configurations belong to the blocklist and can not be applied: api.auth_backend.. Is there another way to enable / access the Airflow API on AWS MWAA? – Babysitter

@Babysitter Did you manage to figure out an alternative way to access Airflow api? – Ruthanneruthe

No, there was no way of going around the AWS restrictions at the time. I do not know if things changed since though. – Babysitter