Snowflake is not showing in the connections dropdown. I am using MWAA 2.0 and the providers are already in the requirements.txt MWAA uses python 3.7 dont know if this can be a thing Requirements.tx...

In Airflow, I have some issues while running multiple tasks, in this case I have 2 DAGs which 1st DAG has 50 tasks to be done, and 2nd is 5 tasks to be done as follows: dynamic_Task_Concurrency.py ...

To set up the connections and variables in airflow i use a DAG, we do this inorder to setup airflow fast in case we have to setup everything again fast. It does work my connections and variables sh...

When task instances marked with State.REMOVED state in Airflow? When I am dynamically generating tasks using for loop, after execution of one of the tasks other tasks are getting removed and not re...

I am running Airflowv1.10.15 on Cloud Composer v1.16.16. My DAG looks like this : from datetime import datetime, timedelta # imports from airflow import DAG from airflow.operators.python_operator ...

I am using the following docker-compose image, I got this image from: https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml version: "3" x-airflow-comm...

I've uploaded my requirementst.txt to a dag and referenced it when creating my Airflow Environment. Here is what my requirements.txt looks like: apache-airflow==1.10.12 oauth2client==4.1.3 google-a...

Code: from airflow.models import BaseOperator from airflow.utils.decorators import apply_defaults from airflow.providers.google.cloud.hooks.gcs import GCSHook class GCSUploadOperator(BaseOperator...

Need some help in understanding the locking behavior around DagRun scheduling. We noticed that after a few DagRuns the subsequent runs are no longer getting scheduled and notice the following in th...

My Airflow install has stopped working. UI Webserver doesn't come up. I think the problem is related to the following error that appears when I run > airflow initdb sqlalchemy.exc.OperationalErr...

We are trying to run a simple DAG with 2 tasks which will communicate data via xcom. DAG file: from __future__ import print_function import airflow from airflow import DAG from airflow.operators....

Summary of my DAG: I am using SSH Operator to SSH to an EC2 instance and run a JAR file which will connect to multiple DBs. I've declared the Airflow Connection in my DAG file and able to pass th...

I have a workflow where I have two parallel processes (sentinel_run and sentinel_skip) which should run or be skipped based on a condition, and then join together (resolve). I need tasks directly d...

We're using Airflow v2.2.3 on Kubernetes (KubernetesExecutor), our environment requires a DAG pre-customer, and each customer can be in a different timezone. Each DAG should be scheduled in its own...

I do not seem to understand how to import modules into an apache airflow DAG definition file. I would want to do this to be able to create a library which makes declaring tasks with similar setting...

I'm writing tests for airflow dag and running into issue mocking/patching the dag. # dag.py from airflow.models import Variable ENVIRONMENT = Variable.get("environment") # test_dag.py import da...

I am using Airflow through Docker-compose. The same docker-compose.yml has an image of Postgres as well. The config looks like below - postgres: image: postgres:13 ports: - 5432:5432 environme...

I'm creating a dev environment to use airflow for testing. I'm using the docker-compose.yaml file available on Airflow website. I would like to know if it is possible to set my connections and vari...

I've a step in my cicd pipeline that install airflow to test the dags loading. Since this morning has worked well. The steps are the following: - apt update -y - pip install --upgrade pip - pip ...

I forgot my local airflow webserver password, it is possible to reset it? I tried to create a new user, but it say that my account already exists, so I would like to reset the password.

What's the best way to get a SQLAlchemy engine from an Airflow connection ID? Currently I am creating a hook, retrieving its URI, then using it to create a SQLAlchemy engine. postgres_hook = Post...

Using apache airflow, I created some DAGS, some of which do not run on a schedule. I'm trying to find a way that I can trigger a run for a specific DAG from within a Python script. Is this possible...

Airflow scheduler kinda left me scratching my head for the past few days as it backfills dag runs even after catchup=False. My timezone-aware dag has a start date of 13-04-2021 19:30 PST or 14-04-2...

We are running a large number of DAGS, and as a result, thousands of task instances for each calendar date. We have sporadic days that have failed and we are attempting to re-run those dates as eff...

I have a airflow DAG "example_ml.py" which has a task "train_ml_model" and this task is calling/running a python script "training.py". -Dags/example_ml.py -Dags/traini...