I am running Airflow via docker through this image apache/airflow:2.1.0 Please refer to this thread for the initial error I faced. Currently I am able to run my previous existing DAGs. However when...

I want to use airflow DAG to run some jobs. I have scheduled the expression to every 25 mins, like */25 * * * *. for instance, it seems to run, like at 6:25, 6:50, and at 7 as well, but I want to r...

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_fo...

I see that one can trigger_dag with parameters/config key-value pairs using the airflow command line: For Apache Airflow, How can I pass the parameters when manually trigger DAG via CLI? In my ca...

I am trying to configure remote logging with Azure blob. Airflow version: 1.10.2 Python: 3.6.5 Ubuntu: 18.04 Following are the step I did: In $AIRFLOW_HOME/config/log_config.py, I have put REM...

Airflow scheduler kinda left me scratching my head for the past few days as it backfills dag runs even after catchup=False. My timezone-aware dag has a start date of 13-04-2021 19:30 PST or 14-04-2...

We are running a large number of DAGS, and as a result, thousands of task instances for each calendar date. We have sporadic days that have failed and we are attempting to re-run those dates as eff...

I'm testing the use of Airflow, and after triggering a (seemingly) large number of DAGs at the same time, it seems to just fail to schedule anything and starts killing processes. These are the logs...

Does anyone use MWAA in production? We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unknown reasons. Task is in the...

We have 100 of dags which has a prefix with "dag_EDW_HC_*" . we have below command to pause the dag Command: airflow pause dag_id Is there any way we can pause all the 100 dags "dag...

I am currently using Airflow Taskflow API 2.0. I am having an issue of combining the use of TaskGroup and BranchPythonOperator. Below is my code: import airflow from airflow.models import DAG from ...

I am new to airflow, tried to run a dag by starting airflow webserver and scheduler. After I closed the scheduler and airflow webserver, the airflow processes are still running. ps aux | grep airfl...

Is it possible to run an airflow task only when a specific event occurs like an event of dropping a file into a specific S3 bucket. Something similar to AWS Lambda events There is S3KeySensor but ...

I am using airflow cli's backfill command to manually run some backfill jobs. airflow backfill mydag -i -s 2018-01-11T16-00-00 -e 2018-01-31T23-00-00 --reset_dagruns --rerun_failed_tasks The d...

I have a dag that we'll deploy to multiple different airflow instances and in our airflow.cfg we have dags_are_paused_at_creation = True but for this specific dag we want it to be turned on without...

We're running into a permission error when using Airflow, receiving the following error: PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler/2019-12-18/../../../../ho...

I have a little problem, I want to do the typical conditional like setting_x = Variable.get('setting_x') variable = setting_x if setting_x else 0 But since the Airflow model throws an exception...

I am running Airflowv1.9 with Celery Executor. I have 5 Airflow workers running in 5 different machines. Airflow scheduler is also running in one of these machines. I have copied the same airflow.c...

I have an upload folder that gets irregular uploads. For each uploaded file, I want to spawn a DAG that is specific to that file. My first thought was to do this with a FileSensor that monitors the...

I'm running Airflow 1.9.0 with LocalExecutor and PostgreSQL database in a Linux AMI. I want to manually trigger DAGs, but whenever I create a DAG that has schedule_interval set to None or to @once,...

I have an Airflow DAG scheduled to run daily. When I start a backfill for the last month, Airflow will start processing the runs from oldest to newest. As a single run takes a couple of hours, whic...

I try to use configs in dag using "trigger w/config". def execute(**kwargs): dag_run = kwargs['dag_run'] start_date = dag_run.conf['start_dt'] if 'start_dt' in dag_run.conf.keys() el...

Colleagues, we need help. There are two dags Parent and Child, parent has its own schedule, suppose '30 * * * * ', child '1 8-17 * * 1-5', child waits for parent to execute, for example 40 minutes,...

I am new to Apache airflow and DAG. There are total 6 tasks in the DAG (task1, task2, task3, task4, task5, task6). But at the time of running the DAG we are getting the error below. DAG unsupported...

Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider...