I am running Airflowv1.10.15 on Cloud Composer v1.16.16. My DAG looks like this : from datetime import datetime, timedelta # imports from airflow import DAG from airflow.operators.python_operator ...

What's the best way to retry an Airflow operator only for certain failures/exceptions? For example, let's assume that I have an Airflow task which relies on the availability of an external service...

I implemented a few tasks with BashOperator. Ones with "gsutil rm" and "gsutil cp" worked fine. But one with "gcloud alpha firestore export" generates this error: {bash_operator.py:101} INFO - ERR...

I am running Airflow on Google's Cloud Composer. I am using the KubernetesPodOperator and would like to mount a google storage bucket to a directory in pod via gcsfuse. It seems like to do this I n...

I'd like to get some clarification on whether Cloud Dataflow or Cloud Composer is the right tool for the job, and I wasn't clear from the Google Documentation. Currently, I'm using Cloud Dat...

I'm following the documentation for Google Cloud Composer to install Python dependencies from PyPI in an environment. I used this command to install the libraries from a requirements file: $ gcloud...

I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them. Example : from src.kuzzle import KuzzleQuer...

I want to get email notifications with Cloud Composer but I am unsure how to do that. How can I configure a Composer environment to send email notifications?

I'm trying to set up a local development environment in VS Code where I'd get code completion for the packages Cloud Composer/Apache Airflow uses. I've been successful so far using a virtual enviro...

In my Composer Airflow DAGs, I have been using the CloudSqlProxyRunner to connect to my Cloud SQL instance. However, after updating Google Cloud Composer from v1.18.4 to 1.18.6, my DAG started to e...

Given that GCP Cloud Composer is running with GKE/GCE, is it auto scaling? Now I have 3 nodes in the cluster to support say, 100 DAGs. Later, if I have about 300 DAGs, will it scale up itself (w/ c...

I'm running a DAG in Google Cloud Composer (hosted Airflow) which runs fine in Airflow locally. All it does is print "Hello World". However, when I run it through Cloud Composer I receive the error...

For a requirement I want to call/invoke a cloud function from inside a cloud composer pipeline but I cant find much info on it, I tried using SimpleHTTP airflow operator but I get this error: [2021...

I have to add labels to the bigquery tables. I am aware of doing it through BigQuery UI but how can it be done through airflow operators. Use case : For billing and searching purpose. As multiple t...

I'm trying to run GKEStartPodOperator/KubernetesPodOperator tasks in a Composer 2 environment, which makes use of a GKE cluster in autopilot mode. We have an existing Composer 1 environment with a ...

I have a directory structure as such: airflow_dags ├── dags │ └── hk │ └── hk_dag.py ├── plugins │ └── cse │ └── operators.py │ └── cse_to_bq.py └── test └── dags └── dag_test.py In the G...

Currently working on setting up alerts for long running tasks in Airflow. To cancel/fail the airflow dag I've put "dagrun_timeout" in the default_args, and it does what I need, fails/erro...

I am trying to setup a Google Cloud Platform connection in Google Cloud Composer using the service account key. So I created a GCS bucket and put the service account key file in the bucket. The key...

So I want to create a dag that creates dynamic tasks based on messages passed from the previous dag. ex:- I have a master dag and a slave dag. from master dag, I pass a list of countries ex: [us,uk...

I have several multi-purpose shell scripts stored in .sh files. My intention is to build a few Airflow DAGs on Cloud Composer that will leverage these scripts. The DAGs would be made mostly of Bash...

We recently met a known issue on airflow: Airflow "This DAG isnt available in the webserver DagBag object " Now we used a temporary solution to restart whole environment by changing ...

I'm using the BigQueryOperator extenstively in my Airflow DAGs on Google Cloud Composer. For longer queries, it's better to put each query in its own .sql file rather than cluttering up the DAG wi...

We have relatively complex dynamic DAG as part of our ETL. DAG contains hundreds of transformations and it is created programmatically based on set of yaml files. It is changed through time: new ta...

I am running Airflow (1.10.9) through Cloud Composer (1.11.1) on GCP. Whenever I update a DAG's code I can see the updated code refreshed in the Airflow GUI but for at least 10 minutes the DAG's ta...

When I need to restart the webserver locally I do: ps -ef | grep airflow | awk '{print $2}' | xargs kill -9 airflow webserver -p 8080 -D How can I do this on Google Composer? I don't see an opti...