I am new to Docker so I have followed the official guide here for running Airflow with it-https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html When I start Airflow all services w...

i am running the airflow pipeline but codes looks seems good but actually i'm getting the airflow.exceptions.AirflowException: Cycle detected in DAG. Faulty task: can u please help to resolve this...

Is it possible to run an airflow task only when a specific event occurs like an event of dropping a file into a specific S3 bucket. Something similar to AWS Lambda events There is S3KeySensor but ...

In my DAG, I've got a task flow like this: ... >> EmrAddStepsOperator >> EmrStepSensor A success of EmrAddStepsOperator means, "I was able to tell EMR to start." A failure of EmrStep...

I am using airflow cli's backfill command to manually run some backfill jobs. airflow backfill mydag -i -s 2018-01-11T16-00-00 -e 2018-01-31T23-00-00 --reset_dagruns --rerun_failed_tasks The d...

My company uses git-sync to sync zipped dags to airflow. We use airflow helm charts to deploy airflow. I wonder if I can let airflow only pick up zipped dags in a specific folder such as dags-dev i...

I have a dag that we'll deploy to multiple different airflow instances and in our airflow.cfg we have dags_are_paused_at_creation = True but for this specific dag we want it to be turned on without...

I am deploying airflow latest release using Docker, everything works fine until I point the database to an external Postgres Database. As soon as the database is pointed to an external database the...

I am using the official docker-compose file of airflow to spin it up. Some of my containers seem unhealthy: 34d8698d67e7 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 28 m...

I am using airflow for my data pipeline project. I have configured my project in airflow and start the airflow server as a backend process using following command airflow webserver -p 8080 -D True...

I am getting started with Airflow and trying to use the KubernetesPodOperator, but I am having trouble with downloading images from private registries. I did some research but I couldn't find an an...

I have Airflow jobs, which are running fine on the EMR cluster. what I need is, let's say if I have a 4 airflow jobs which required an EMR cluster for let's say 20 min to complete the task. why not...

We're running into a permission error when using Airflow, receiving the following error: PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler/2019-12-18/../../../../ho...

I have a DAG in airflow and for now it is running each hour (@hourly). Is it possible to have it running each 5 minutes ?

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggre...

I made a small change to an existing workflow, and it has broken airflow. Here is the code: dag_name = platform + "_" + report['table'] dag = DAG( dag_name, catchup=True, default_args=default_...

I am a new user of Airbnb's open source workflow/datapipeline software airflow. There are dozens of default example dags after the web UI is started. I tried many ways to remove these dags, but I'v...

I have a little problem, I want to do the typical conditional like setting_x = Variable.get('setting_x') variable = setting_x if setting_x else 0 But since the Airflow model throws an exception...

I am using docker compose with bitnami's airflow image as well as minio. I can get airflow to talk to AWS S3, but when I try to substitute Minio I am getting this error: File "/opt/bitnami/air...

We have an Airflow deployment with Celery executors. Many of our DAGs require a local processing step of some file in a BashOperator or PythonOperator. However, in our understanding the tasks of ...

I need to reference a variable that's returned by a BashOperator. In my task_archive_s3_file, I need to get the filename from get_s3_file. The task simply prints {{ ti.xcom_pull(task_ids=submit_fil...

I am using airflow to trigger jobs on databricks. I have many DAGs running databricks jobs and I whish to have to use only one cluster instead of many, since to my understanding this will reduce th...

I'm trying to access external files in a Airflow Task to read some sql, and I'm getting "file not found". Has anyone come across this? from airflow import DAG from airflow.operators.python_operato...

I haven't been able to move common code outside of the dag directory that airflow uses. I've looked in the airflow source and found imp.load_source. Is it possible to use imp.load_source to load ...

My questions : What is a good directory structure in order to organize your dags and tasks? (the dags examples show only couple of tasks) I currently have my dags at the root of the dags folder a...