Why is there a DAG named 'airflow_monitoring' automatically generated in Cloud Composer?

Question

When creating an Airflow environment on GCP Composer, there is a DAG named airflow_monitoring automatically created and that comes back even when deleted.

Why? How to handle it? Should I copy this file inside my DAG folder and resign myself to make it part of my code? I noticed that each time I upload my code it stops the execution of this DAG as it could not be found inside the DAG folder until it magically reappears.

I have already tried deleting it inside the DAG folder, delete the logs, delete it from the UI, all of this at the same time etc.

Mcmahan

Answer 1

The airflow_monitoring DAG is a per-environment liveness prober/healthcheck that is used to populate the Cloud Composer monitoring metric environment/healthy. It is an indicator for the general overall health of your environment, or more specifically, its ability to schedule DAGs and run tasks. This allows you to use Google Cloud Monitoring features such as metric graphs, or setting alerts when your environment becomes unhealthy.

You can find more information about the metric on the GCP Metrics List, and can explore the metric in Cloud Monitoring under the following:

• Resource type: Cloud Composer Environment
• Metric: Healthy

This is a Composer-managed DAG and uses very minimal resources from your environment. Ideally, you should leave it untouched, as it has little to no effect on anything else running in your environment.

Buffet

I set up a minimum environment in our org as we are a startup and not ingesting a crazy amount of data currently. Why should I gift ~300€/month (~10€/day) to Google for something I don't want to monitor? This is not a critic to you, but I'm trying to understand if this DAG is forever vital or not? – Peridotite

@Peridotite ~300€/month is the minimal cost for using GCP Composer. It has nothing to do with the airflow_monitoring dag. If you would like to pay less you can launch Apache Airflow on Google Compute Engine which will cost you nearly ~30€/month – Commutative