Schedule a DAG in airflow to run every 5 minutes

Question

I have a DAG in airflow and for now it is running each hour (@hourly). Is it possible to have it running each 5 minutes ?

Klayman

Answer 1

Yes, here's an example of a DAG that I have running every 5 min:

dag = DAG(dag_id='eth_rates',
          default_args=args,
          schedule_interval='*/5 * * * *',
          dagrun_timeout=timedelta(seconds=5))

schedule_interval accepts a CRON expression: https://en.wikipedia.org/wiki/Cron#CRON_expression

Centimeter

Answer 2

Airflow 2 (I'm using 2.4.2) supports timedelta for scheduling DAGs on a particular cadence (hourly, every 5 minutes, etc.) so you can put:

schedule_interval = timedelta(minutes=5)

Tigress

Answer 3

The documentation states:

Each DAG may or may not have a schedule, which informs how DAG Runs are created. schedule_interval is defined as a DAG arguments, and receives preferably a cron expression as a str, or a datetime.timedelta object.

When following the provided link for CRON expressions it appears you can specify it as */5 * * * * to run it every 5 minutes.

I'm not familiar on the matter, but this is what the documentation states.

Receptive

5 0 0 0 0 is invalid. I think you mean */5 * * * * – Redress