Airflow wrong execution_date for timezone aware DAGs

Question

We're using Airflow v2.2.3 on Kubernetes (KubernetesExecutor), our environment requires a DAG pre-customer, and each customer can be in a different timezone.

Each DAG should be scheduled in its own timezone at midnight, and I saw that it can be achieved using Airflow's timezone aware DAGs

so configuring timezone aware start_date for each DAG worked and made each DAG to execute in its own timezone midnight:

start_date_utc = (datetime.now() - timedelta(days=2)).replace(
        hour=0, minute=0, second=0, microsecond=0)

timezone = pendulum.timezone(get_customer_timezone(customer))
START_DATE = start_date_utc.replace(tzinfo=timezone)

default_args = {
        "owner": "owner",
        "depends_on_past": False,
        "start_date": START_DATE,
}

dag = DAG(
    dag_id,
    schedule_interval="0 0 * * *",
    default_args=default_args,
    tags=[cusotmer_name]
)

date = '{{ execution_date | ds }}'

operator_args = {
    "customer_date": date,
}

My problem is that both the jinja template and dag_run the execution_date (dag_run.logical_date) is still in UTC, and was not adjusted based on the DAGs timezone.

That causes unexpected behavior when running DAGs in different timezones tho execution_date of DAGs with timezone offset earlier than UTC is wrong (2 days before and not 1)

I need some advice please on how can we change the execution_date based on the DAGs timezone

thanks

Psychopharmacology

For example: {{ execution_date.in_timezone('Europe/Amsterdam') ? – Cutworm

@Elad that's cool, but each DAG have its own timezone so I'll need to access the dag's timezone from within the template – Psychopharmacology

You have it in dag.timezone – Cutworm

thanks @Elad , I was able to solve it as you suggested using: {{ dag_run.logical_date.astimezone(dag.timezone) }} in template, I wondered if the execution_date for timezone aware dags can be in DAGs timezone, cause currently as far as I can see in docs it's only in UTC ... – Psychopharmacology

Answer 1

The value of execution_date is in UTC. To convert to different time zone you can do:

{{ execution_date.in_timezone('Europe/Amsterdam') }}

If you set timezone in your DAG you can access it from dag.timezone and use it Jinja.

{{ dag_run.execution_date.astimezone(dag.timezone) }}

You already noticed that execution_date is deprecated macro so you should use logical_date as:

{{ dag_run.logical_date.astimezone(dag.timezone) }}

Cutworm

Answer 2

https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html#templates

import pendulum

local_tz = pendulum.timezone("Europe/Amsterdam")
local_tz.convert(logical_date)

Hussite