I am trying to add a running instance of MinIO to Airflow connections, I thought it should be as easy as this setup in the GUI (never mind the exposed credentials, this is a blocked of environment ...

I am running Airflow via docker through this image apache/airflow:2.1.0 Please refer to this thread for the initial error I faced. Currently I am able to run my previous existing DAGs. However when...

I have a very simple DAG with two tasks, like following: default_args = { 'owner': 'me', 'start_date': dt.datetime.today(), 'retries': 0, 'retry_delay': dt.timedelta(minutes=1) } dag = DAG( ...

I know there is a way to delete a DAG. But is it possible to delete a DAG run with a specific run_id? Something like: airflow delete_dag_run <dag_id> <run_id>

With the new Dynamic Task Mapping feature, it's possible to use the expand function and create a variable number of tasks based on the output of a previous task. The problem is that, as shown in th...

I'm trying to migrate all the existing airflow connections to a new airflow. I was looking at the cli options airflow connections --help, it gives an option to list but doesn't give an option to ...

I want to use airflow DAG to run some jobs. I have scheduled the expression to every 25 mins, like */25 * * * *. for instance, it seems to run, like at 6:25, 6:50, and at 7 as well, but I want to r...

I am a newbie to Airflow and struggling with BashOperator. I want to access a shell script using bash operatory in my dag.py. I checked: How to run bash script file in Airflow and BashOperator do...

I'm trying to run a docker container via Airflow but getting Permission Denied errors. I have seen a few related posts and some people seem to have solved it via sudo chmod 777 /var/run/docker.sock...

I'm on MacOS Mojave, Python 3.8.3, and pip 23.2. I made a py_env, activated it, and ran pip install apache-airflow==2.6.3' \ --constraint "https://raw.githubusercontent.com/apache/airflow/con...

I tried below code but still i am getting issue from airflow.models DagModel def get_latest_execution_date(**kwargs): session = airflow.settings.Session() f = open("/home/Insurance/Insuranc...

We have an automated upload system for our DAG's to MWAA. Is there a way to have the requirements.txt in the S3 bucket be automatically set to its latest version? I cannot find an option for this i...

I have an AWS MWAA Airflow v2.0.2 instance running. I would like to have access to the Airflow API. Is this something supported currently? Planned for future releases? There is no mention of this i...

Is it possible to run an R script as an airflow dag? I have tried looking online for documentation on this and am unable to do so. Thanks

I'm trying to install Airflow with docker on my desktop. When I run curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.2.3/docker-compose.yaml' like the internet says, I get this: Invoke-W...

I'm trying to use airflow to define a specific workflow that I want to manually trigger from the command line. I create the DAG and add a bunch of tasks. dag = airflow.DAG( "DAG_NAME", start_d...

We are planning to switch from managing airflow ourselves to Managed Apache Airflow services of AWS. Our original dags use some custom environment variables that need to be set in Managed airflow a...

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_fo...

Is it possible to get the actual start time of a dag in Airflow? By start time I mean the exact time the first task of a dag starts running. I know I can use macros to get the execution date. If t...

I have a list of dags that are hosted on Airflow. I want to get the name of the dags in a AWS lambda function so that I can use the names and trigger the dag using experimental API. I am stuck on g...

I see that one can trigger_dag with parameters/config key-value pairs using the airflow command line: For Apache Airflow, How can I pass the parameters when manually trigger DAG via CLI? In my ca...

I am trying to configure remote logging with Azure blob. Airflow version: 1.10.2 Python: 3.6.5 Ubuntu: 18.04 Following are the step I did: In $AIRFLOW_HOME/config/log_config.py, I have put REM...

How to configure the Airflow dag to execute at specified time on daily basis no matter what happens, something exactly like crons. I know that similar behaviour could be obtained using TimeSensor,...

I found out there are lot of ways to store it as variables, hooks and other ways using encryption. I would like to know what's the best way to do it.

I just started with Airflow. I want to set up a DAG in a loop, where the next DAG starts when the previous DAG is completed. Here is the work flow that I want to achieve: list_of_files = [......] ...