With the new Dynamic Task Mapping feature, it's possible to use the expand function and create a variable number of tasks based on the output of a previous task. The problem is that, as shown in th...

In Airflow, I have some issues while running multiple tasks, in this case I have 2 DAGs which 1st DAG has 50 tasks to be done, and 2nd is 5 tasks to be done as follows: dynamic_Task_Concurrency.py ...

I am looking to write docker compose file to locally execute airflow in production similar environent. For older airflow v1.10.14, docker compose is working fine. But same docker compose is not wor...

Does anyone use MWAA in production? We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unknown reasons. Task is in the...

Requirement: To connect and execute the RScript with connection from R to SnowflakeDB I am trying to set up a Docker image that can communicate with a Snowflake database through R (either using ROD...

Getting error while running the airflow DAG code in visual studio code. Error ImportError: cannot import name 'DummyOperator' from 'airflow.operators' (c:\Users\10679196\AppData\Local\Programs\Pyth...

I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them. Example : from src.kuzzle import KuzzleQuer...

I am passing some run-time DAG params/config to a PythonOperator in a very similar way to these Airflow docs: def print_x(x): print(f"x is {x}") with DAG( "print_x", start_...

I have a task in which the output is a dictionary with a list value in each key @task(task_id="gen_dict") def generate_dict(): ... return output_dict # output look like this {"A&q...

Problem : New DAGs or changes to existing DAGs are not showing up on Airflow web server to use in the App. For example. Suppose I add a new DAG in the DAG directory: What does work: If I run $ air...

I am currently attempting to setup LDAP integration with an existing LDAP server in Airflow. In the past, I have attempted making a cacert (ldap_ca.crt) and have followed this guide and this guide....

I am trying to pass the params in postgres operator, in a dynamic way. There are two tasks in order to refresh the metadata, get list of id (get_query_id_task) pass the list of ids to get and exe...