GKE autoscaling doesn't scale down

Question

We use GKE (Google Kubernetes Engine) to run Airflow in a GCC (Google Cloude Composer) for our data pipeline.

We started out with 6 nodes, and realised that the costs spiked, and we didn't use that much CPU. So we thought that we could lower the max, but also enable auto-scaling.

Since we run the pipeline during the night, and only smaller jobs during the day we wanted to run auto-scaling between 1-3 nodes.

So on the GKE node pool we enabled autoscaling, but not on the GCE instance group as they recommend. However, we get this:

Why is this?

Below is a graph of our CPU utilisation over the past 4 days:

We never pass 20% usage, so why doesn't it scale down?

This morning we manually scaled it down to 3 nodes..

Salesperson

Do any of these troubleshooting items apply? – Dunseath

cloud.google.com/kubernetes-engine/docs/how-to/… – Pistol

Answer 1

Cloud Composer does not yet (as of 2019/08/26) support the GKE cluster autoscaler, because the cluster autoscaler makes scaling decisions based on the resource requests of Pods, as well as how many Pods are in the unschedulable state (more information here). Composer deploys a fixed number of Pods, which means the autoscaling mechanism doesn't force any scaling action unless you yourself are deploying your own workloads into the cluster.

Autoscaling is also difficult to do because the actual resource usage that an Airflow worker or scheduler depends on how many DAGs you upload (into GCS, in Composer's case), meaning there is no accurate estimate of how much CPU/memory your Airflow processes will use. That means you have no idea how to decide on Pod resource requests of the Airflow Pods.

---

In the absence of autoscaling, there are still many options for dynamic resource allocation. For example, you can use KubernetesPodOperator to deploy Pods with resource requests into a different Kubernetes cluster that does have autoscaling enabled. Alternatively, you can use the GCE operators to add instances to your cluster before launching more resource-heavy workloads.

Brittni

Thanks! This answers my question. A bit more complex than we hoped for. I think we lean towards using the GCE operators and adding instances during our nightly runs. – Gravestone

I tried to use the GCE operators to stop and start instances in my cluster, but it did not work as hoped for. When I stopped an instance using the API it automatically started again after a few minutes. I guess that is due to that the Node pool is set to 3 instances, and as soon as it realise that one of the nodes has stopped it automatically starts it again. Was that the way you meant we could use that operator @hexacyanide? – Gravestone

I did some write up on composer autoscaling link.medium.com/AMUlwUIkD0 there are some calculations for the resource request as well. Do check it out – Syntactics

Answer 2

The first thing I want to mention is that the scale-down process is triggered when exist underutilized nodes in the cluster. In this context, "underutilized" is not only related with CPU usage only so your reasoning is not completely right.

As documentation says, the condition is that the sum of cpu and memory requests of all pods running on this node is smaller than the utilization threshold defined for the Autoscaler. Then, "if a node is unneeded for more than 10 minutes, it will be terminated". See this for more information.

Also it is important to know that there are some other factors that could prevent the scale-down process, as for instance the node auto-provisioning limits. Check this for more info about pods that can prevent cluster autoscaler from removing a node.

Ribonuclease

Answer 3

So, it's a bit strange that you say "it's not supported" its k8s. Enable GKE cluster auto-scaler as specified here:

gcloud container clusters update [CLUSTER_NAME] --enable-autoscaling \
    --min-nodes 1 --max-nodes 10 --zone [COMPUTE_ZONE] --node-pool default-pool 

this is on default node pool, if you created a new pool, use that one.

Go to your airflow-worker deployment and add in this deployment just after name: airflow-worker or ports:

resource:
  requests:
    cpu: 400m

Then after this, autoscale you deployment like this:

kubectl autoscale deployment airflow-worker --cpu-percent=95 --min=1 --max=10 -n <your namespace>

In my case it works like a charm, it scaling both nodes and pods for my airflow-worker deployment.

PoC:

$ kubectl get hpa -n  <my-namespace> -w   

• NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
• airflow-worker Deployment/airflow-worker /95% 1 3
  0 13s
• airflow-worker Deployment/airflow-worker 20%/95% 1 10 1 29m
• airflow-worker Deployment/airflow-worker 27%/95% 1 10 2 29m
• airflow-worker Deployment/airflow-worker 30%/95% 1 10 3 29m
• airflow-worker Deployment/airflow-worker 53%/95% 1 10 3 29m
• airflow-worker Deployment/airflow-worker 45%/95% 1 10 3 34m
• airflow-worker Deployment/airflow-worker 45%/95% 1 10 3 34m
• airflow-worker Deployment/airflow-worker 28%/95% 1 10 2 34m
• airflow-worker Deployment/airflow-worker 32%/95% 1 10 2 35
• airflow-worker Deployment/airflow-worker 37%/95% 1 10 2 43m
• airflow-worker Deployment/airflow-worker 84%/95% 1 10 1 43m
• airflow-worker Deployment/airflow-worker 39%/95% 1 10 1 44m
• airflow-worker Deployment/airflow-worker 29%/95% 1 10 1 44m

You can see that after a while it`s scaling down to 1pod. Same as the node pool is scaling down to 4 nodes instead of 5-6-7..

Lightness

Hmm nice, I'll look into this and try it out to see if it works! Thanks! – Gravestone

So, how was it? – Lightness

At this time, you can configure autoscaling if you are willing to modify the Kubernetes configuration. However, Composer cannot guarantee that this configuration will not be overwritten on environment update or upgrade. – Brittni