I need to create a custom airflow operator which I should be able to use in airflow template(Written in python) which is running in cloud composer... If I create custom airflow operator how can I ...

After some research and testing, we have decided to start using Google Cloud Composer. Since our current DAGs and tasks are relatively small, and don't require the server to run continuously, I am ...

In order to reduce the billing associated of running the google-cloud-composer, I am wondering about the possibility to turn off the VM instances that run the Virtual Environment at certain hours. ...

As title, we can set pypi packages in requirements.txt file and use the command gcloud beta composer environments update env_name --update-pypi-packages-from-file requirements.txt --location locat...

I am trying to run the example in Google Cloud Composer documentation on and I find issues, mainly two: the environment variables, when created via the gcloud command line or the web interface, d...

According to the documentation here: https://cloud.google.com/composer/docs/release-notes Composer (GCP's Airflow) is supposed to be available for Python 3 in the console. However, I am seeing no ...

We are using Google Composer for workflow orchestration, randomly we are getting An internal server error occurred while authorizing your request. Error code 28 message while opening the web UI. We...

My Cloud Composer managed Airflow got stuck for hours since I've canceled a Task Instance that was taking too long (Let's call it Task A) I've cleared all the DAG Runs and task instances, but the...

As for the documentation, Google Cloud Composer airflow worker nodes are served from a dedicated kubernetes cluster: I have a Docker contained ETL step that I would like to run using airflow, pr...

I'm using Google Cloud Composer (managed Airflow on Google Cloud Platform) with image version composer-0.5.3-airflow-1.9.0 and Python 2.7, and I'm facing a weird issue : after importing my DAGs, th...

Airflow allows you to put dependencies (external python code to the dag code) that dags rely on in the dag folder. this means any components/members or classes in those external python code is avai...

To separate bigquery queries from the actual code I want to store the sql in a separate file and then read it from the python code. I have tried to add the file in the same bucket as the DAGs and a...