I am running Airflowv1.9 with Celery Executor. I have 5 Airflow workers running in 5 different machines. Airflow scheduler is also running in one of these machines. I have copied the same airflow.c...

I'm trying to create a simple DAG workflow on Apache Airflow where data is updated to my localhost PostgreSQL database. Can someone please tell me why my DAG fails and I get these errors in the log...

I would like to create a conditional task in Airflow as described in the schema below. The expected scenario is the following: Task 1 executes If Task 1 succeed, then execute Task 2a Else If Task...

We have many AWS connection string in apache airflow and anyone can see our access keys and secret keys in airflow webserver connections section. How to hide or mask sensitive data in airflow webse...

I'm new to Apache Airflow. I have been modifying the schedule_interval and replacing the python script, each time I want to change the execution time. Can I change the DAG schedule without uploadi...

In my actual DAG, I need to first get a list of IDs and then for each ID run a set of tasks. I have used Dynamic Task Mapping to pass a list to a single task or operator to have it process the list...

I'm using Airflow 2.0.0, and my tasks are sporadically being killed "externally" after running for a few seconds or minutes. The tasks usually run successfully (both for manual task initi...

I have a dag which checks for new workflows to be generated (Dynamic DAG) at a regular interval and if found, creates them. (Ref: Dynamic dags not getting added by scheduler ) The above DAG is wor...

I've got dag_prime and dag_tertiary. dag_prime: Scans through a directory and intends to call dag_tertiary on each one. Currently a PythonOperator. dag_tertiary: Scans through the directory pas...

I'm trying to setup Postgres as backend database for my Airflow installation. I followed what the documentation suggests. Particularly, I created a specific user for Airflow in Postgres, a specific...

Getting error while running the airflow DAG code in visual studio code. Error ImportError: cannot import name 'DummyOperator' from 'airflow.operators' (c:\Users\10679196\AppData\Local\Programs\Pyth...

I have an upload folder that gets irregular uploads. For each uploaded file, I want to spawn a DAG that is specific to that file. My first thought was to do this with a FileSensor that monitors the...

I would like to build an airflow graph like this: ┌───► task_2_1 ─────► task_2_1 │ │ │ task 1 ───────────► task_2_2 ─────► task_3_2 │ │ │ │ └──► task_2_N ─────► task_3_N Where the number ...

I'm running Airflow 1.9.0 with LocalExecutor and PostgreSQL database in a Linux AMI. I want to manually trigger DAGs, but whenever I create a DAG that has schedule_interval set to None or to @once,...

I've been asked to learn Docker so that we can eventually setup our web scraping scripts in Apache Airflow Docker + Kubernetes clusters. If I am using these terms the wrong way, it's because I am a...

I have a task that I'll call final that has multiple upstream connections. When one of the upstreams gets skipped by ShortCircuitOperator this task gets skipped as well. I don't want final task to ...

I have an Airflow DAG scheduled to run daily. When I start a backfill for the last month, Airflow will start processing the runs from oldest to newest. As a single run takes a couple of hours, whic...

I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch. I am getting the followi...

I am using Apache Airflow where in one of our DAG's task we are using Kubernetes Pod Operator. This is being done to execute one of our application process in a kubernetes pod. The Kubernetes pod o...

I try to use configs in dag using "trigger w/config". def execute(**kwargs): dag_run = kwargs['dag_run'] start_date = dag_run.conf['start_dt'] if 'start_dt' in dag_run.conf.keys() el...

Colleagues, we need help. There are two dags Parent and Child, parent has its own schedule, suppose '30 * * * * ', child '1 8-17 * * 1-5', child waits for parent to execute, for example 40 minutes,...

I have file called properties.json which looks like { "FOO": { "var1": "apple", "settings": { "more_settings": { "var2": "pear", "var3": 123 } } }, "BAR": "FOO" } and I'm trying to imp...

I am calling weather API using Python script but the airflow task fails with error Negsignal.SIGSEGV. The Python script to call the weather API work fine when ran outside Airflow. DAG from airflow ...

I am running Apache Airflow on docker. I want to install an airflow provider package for spark. My docker compose yml file looks like this. I want to add Spark as my connection type when i try to c...

Learning apache airflow and I'm trying to create a new connection type the correct way, but it still doesn't show up. I am working within a virtual environment on WSL2 through VS Code, and my termi...