Time-based scaling with Kubernetes CronJob: How to avoid deployments overriding minReplicas

Question

I have a HorizontalPodAutoscalar to scale my pods based on CPU. The minReplicas here is set to 5:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-web
  minReplicas: 5 
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

I've then added Cron jobs to scale up/down my horizontal pod autoscaler based on time of day:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: production
  name: cron-runner
rules:
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["patch", "get"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: cron-runner
  namespace: production
subjects:
- kind: ServiceAccount
  name: sa-cron-runner
  namespace: production
roleRef:
  kind: Role
  name: cron-runner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cron-runner
  namespace: production
---

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-up-job
  namespace: production
spec:
  schedule: "56 11 * * 1-6"
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-up-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":8}}'
          restartPolicy: OnFailure
----
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: django-scale-down-job
  namespace: production
spec:
  schedule: "30 20 * * 1-6"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 0 # Remove after successful completion
  failedJobsHistoryLimit: 1 # Retain failed so that we see it
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cron-runner
          containers:
          - name: django-scale-down-job
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - kubectl patch hpa myapp-web --patch '{"spec":{"minReplicas":5}}'
          restartPolicy: OnFailure

This works really well, except that now when I deploy it overwrites this minReplicas value with the minReplicas in the HorizontalPodAutoscaler spec (in my case, this is set to 5)

I'm deploying my HPA using kubectl apply -f ~/autoscale.yaml

Is there a way of handling this situation? Do I need to create some kind of shared logic so that my deployment scripts can work out what the minReplicas value should be? Or is there a simpler way of handling this?

Absolutism

Hello, I'm having a difficulty to understand your question. Could you tell the whole scenario for the time based HPA scaling of yours? What exactly you mean by overwriting the minReplicas (as it should looking at your example)? You should create HPA that would be encompassing both situation (high load/low load) in a single HPA basing on the available metrics. – Dentist

Ah sorry - I've added the HPA config which should hopefully make it clearer – Absolutism

If you would like to avoid overwriting the value of replicas in your Deployment (each new create/apply) you could try to update your Deployment with JSON merge patch. Is this what you are looking for? – Dentist

Ah...you mean each deployment you use a merge patch instead of apply, then this will only update values that have explicitly changed? Will give that a go and revert back if so, sounds smart. – Absolutism

I've had a go with this and clarified the problem. Looks like it's definitely the HPA minReplicas value that's overwriting the one set by the CronJob (as opposed to the replicas in the Deployment). I tried using JSON merge to deploy the HPA (kubectl patch -f autoscale.yaml --type=merge -p "$(cat autoscale.yaml)") and it didn't work. Any other ideas? – Absolutism

Hello, I think we've had a bit of misunderstanding. I thought you are not changing the HPA resource but only the tag/version of the container during the upgrade process (in the Deployment). That's why I suggested going with the JSON merge patch for it, where you just add the changes to it and not whole resource. Please take a look on the answer I've posted. – Dentist

Answer 1

I think you could also consider the following two options:

---

Use helm to manage the life-cycle of your application with lookup function:

The main idea behind this solution is to query the state of specific cluster resource (here HPA) before trying to create/recreate it with helm install/upgrade commands.

• Helm.sh: Docs: Chart template guide: Functions and pipelines: Using the lookup function

I mean to check the current minReplicas value each time before you upgrade your application stack.

---

Manage the HPA resource separately to application manifest files

Here you can handover this task to a dedicated HPA operator, which can coexist with your CronJobs that adjust minReplicas according specific schedule:

• Banzaicloud.com: Blog: K8S HPA Operator

Dentist

Great answer. I haven't had chance to try the approaches yet, but will accept for now. Very helpful – Absolutism

Answer 2

How to scale down CroneJob not deleting. is there any command like this?

kubectl scale cronjob tms-cron --replicas=0 -n develop

My cronjob yaml is here

--
apiVersion: batch/v1
kind: CronJob
metadata:
  name: tms-cron
  namespace: develop
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          imagePullSecrets:
            - name: s-jenkins
          containers:
            - name: tms-cronjob
              image: docker-c2.example.com/netcore/netcore-v:v0.BUILDVERSION
              imagePullPolicy: IfNotPresent
              envFrom:
                - secretRef:
                    name: netcore-secrets
                - configMapRef:
                    name: netcore-configmap
              command:
                - php
              args:
                - artisan
                - schedule:run
          restartPolicy: OnFailure

Landlady