Get List of all the dags in python

Question

I have a list of dags that are hosted on Airflow. I want to get the name of the dags in a AWS lambda function so that I can use the names and trigger the dag using experimental API. I am stuck on getting the names of the dag. Any help would be appreciated

Stunk

can you please provide the version of Airflow you're using, and anything you have tried so far. – Admeasurement

The question is how to list dags using python. Unless the question is to fins it via APi or CLI? – Talanian

I could not find any endpoint in the Airflow Experimental API (deprecated as of version 2.0) for listing dags (Source: airflow.apache.org/docs/apache-airflow/stable/…). There is one endpoint in the stable API which can be found here: airflow.apache.org/docs/apache-airflow/stable/… (I think it's /api/v1/dags?<query_parameters_here> but I have not tried it myself). You might also have to configure appropriate authentication settings in airflow.cfg before using the API. – Orebro

Answer 1

from airflow.models import DagBag
dag_ids = DagBag(include_examples=False).dag_ids
for id in dag_ids:
   print(id)

Intolerant

thanks - the question specifically asks for Python API and your answer is the only one that actually uses it – Lannylanolin

Answer 2

Since Airflow 2.0, the airflow list_dags command is now:

airflow dags list [-h] [-o table, json, yaml, plain] [-S SUBDIR] [-v]

with the following named arguments:

-o, --output

• Possible choices: table, json, yaml, plain
• Output format. Allowed values: json, yaml, plain, table (default: table)
• Default: “table”

-S, --subdir

• File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
• Default: “[AIRFLOW_HOME]/dags”

-v, --verbose

• Make logging output more verbose
• Default: False

See:

• Airflow CLI changes in 2.0
• Command Line Interface and Environment Variables Reference

Pelmas

Answer 3

All Airflow-CLI commands for the various various are listed on this URL - https://airflow.apache.org/docs/apache-airflow/stable/usage-cli.html

In the latest version you could do

airflow list_dags

Primatology

Answer 4

You can first connect with the backend database, By default airflow using SQLite. Then you can check the DAGs status from table dag using columns is_active and is_paused

e.g. airflow=# SELECT dag_id FROM dag WHERE is_active=TRUE AND is_paused=FALSE;

Skirr

I wish you added some steps on how to connect to the backend database. – Horrible

Answer 5

This command will shows all DAGS include disabled dags as well

airflow dag_list

Neral

Answer 6

This way works slowly, because it probably needs to parse all python files to add DAGs to DagBag:

from airflow.models import DagBag
dag_ids = DagBag(include_examples=False).dag_ids
for id in dag_ids:
   print(id)

In my case on PROD environment it would take 20 seconds to parse all DAGs. Does anyone know better way maybe read DAGs from database? I was trying to do it by passing parameter

read_dags_from_db=True

to DagBag, but it returns empty list of Dags.

Wedded