Composer auto scaling?

Question

Given that GCP Cloud Composer is running with GKE/GCE, is it auto scaling?

Now I have 3 nodes in the cluster to support say, 100 DAGs.

Later, if I have about 300 DAGs, will it scale up itself (w/ celery workers)?

Lagunas

Do you have any idea whether cloud composer supports auto scaling or not? I was creating infrastructure using Terraform but I don't see any such option of auto-scaling there. Is it still not there? – Refurbish

Answer 1

No autoscaling at this point, but I expect it's in the roadmap.

Astronomer is working on an autoscaling Airflow service using Kubernetes, it'll probably launch around the time Google Composer launches autoscaling... so good times are coming :)

Thermionics

Hey @ryw, do you have any idea whether cloud composer supports auto scaling or not? I was creating infrastructure using Terraform but I don't see any such option of auto-scaling there. Is it still not there? – Refurbish

unfortunately i'm not sure how/when they will support autoscaling. – Thermionics

Answer 2

We currently don't support auto-scaling but it's on our roadmap. You could however manually scale up/down the GKE cluster by updating the nodeCount value.

Fedora

Answer 3

I just published an article showing how to enable autoscale in its underlying kubernetes https://link.medium.com/AMUlwUIkD0.

Basically:

1. Enable autoscale on the node level

2. Apply HorizontalPodAutoscaler to airflow-worker deployment

3. Increase some airflow config parameter to remove the bottleneck

This has been tested in composer-1.7.2 and composer-1.7.5 but might be applicable to other version as well

Do check it out

Dwaynedweck

Answer 4

Cloud Composer brings native support for Environment Scaling since major version 2:

gcloud beta composer composer environments update <ENVIRONMENT_NAME> \
--location <LOCATION> \
--min-workers <WORKERS_MIN> \
--max-workers <WORKERS_MAX>

See Scale Environments

Forborne