Airflow (kubernetes executor) - Scheduling skipped because DAG record was locked

Question

Need some help in understanding the locking behavior around DagRun scheduling.

We noticed that after a few DagRuns the subsequent runs are no longer getting scheduled and notice the following in the logs.

{scheduler_job_runner.py:1426} INFO - DAG dag-test scheduling was skipped, probably because the DAG record was locked.

We are currently running a single scheduler pod. Would like to understand the locking behavior in general and under what scenarios does the lock get removed.

Currently the workaround is to restart the scheduler pod and the lock gets released but this isn't ideal for production. Would like to understand when does the lock get released for a DagRun in general ?

Appreciate any help/pointers.

Aesir

Linking the related issue github.com/apache/airflow/issues/36920 here. For us, restarting the scheduler pods didn't resolve the issue. – None

There's a related issue about it - github.com/apache/airflow/issues/36920 – Sophi

Answer 1

Did you tried to clean all DagRuns. In UI go to Browse -> DAG Runs -> Select all dag runs -> Actions -> Delete/.

Strategist