Prevent multiple cron running in nest.js on docker

Question

In docker we have used deploy: replicas: 3 for our microservice. We have some Cronjob & the problem is the system in running all cronjob is getting called 3 times which is not what we want. We want to run it only one time. Sample of cron in nest.js :

  @Cron(CronExpression.EVERY_5_MINUTES)
  async runBiEventProcessor() {
    const calculationDate = new Date()
    Logger.log(`Bi Event Processor started at ${calculationDate}`)

How can I run this cron only once without changing the replicas to 1?

Kieffer

Answer 1

This is quite a generic problem when cron or background job is part of the application having multiple instances running concurrently.

There are multiple ways to deal with this kind of scenario. Following are some of the workaround if you don't have a concrete solution:

1. Create a separate service only for the background processing and ensure only one instance is running at a time.

2. Expose the cron job as an API and trigger the API to start background processing. In this scenario, the load balancer will hand over the request to only one instance. This approach will ensure that only one instance will handle the job. You will still need an external entity to hit the API, which can be in-house or third-party.

3. Use repeatable jobs feature from Bull Queue or any other tool or library that provides similar features. Bull will hand over the job to any active processor. That way, it ensures the job is processed only once by only one active processor. Nest.js has wrapper for the same. Read more about the Bull queue repeatable job here.

4. Implement a custom locking mechanism It is not difficult as it sounds. Many other schedulers in other frameworks work on similar principles to handle concurrency.

   • If you are using RDBMS, make use of transactions and locking. Create cron records in the database. Acquire the lock as soon as the first cron enters and processes. Other concurrent jobs will either fail or timeout as they will not be able to acquire the lock. But you will need to handle a few cases in this approach to make it bug-free and flawless.
   • If you are using MongoDB or any similar database that supports TTL (Time-to-live) setting and unique index. Insert the document in the database where one of the fields from the document has unique constraints that ensure another job will not be able to insert one more document as it will fail due to database-level unique constraints. Also, ensure TTL(Time-to-live index) on the document; this way document will be deleted after a configured time.

These are workaround if you don't have any other concrete options.

Corina

If you have any further queries or if you didn't understand a particular part. Let me know; I will edit the answer to make it more clear. – Corina

Bull queue is not skipping repeatable task but only postpones it, so if you have heavy task the queue will become larger and larger with time, for me custom locking mechanism worked best – Favored

Answer 2

There are quite some options here on how you could solve this, but I would suggest to create a NestJS microservice (or plain nodeJS) to run only the cronjob and store it in a shared db for example to store the result in Redis.

Your microservice that runs the cronjob does not expose anything, it only starts your cronjob:

const app = await NestFactory.create(
  WorkerModule,
);
await app.init();

Your WorkerModule imports the scheduler and configures the scheduler there. The result of the cronjob you can write to a shared db like Redis.

Now you can still use 3 replica's but prevent registering cron jobs in all replica's.

Fassett

I don't want to create a microservice for this. Need another way to do it. – Kieffer

Then the same approach, use some shared DB and store the result of the cronjob with a timestamp. If it is present, skip running. – Fassett

Answer 3

If you are using Docker, you can set one of the replicas as a distinct service and assign an additional environment variable as a flag to execute the cron jobs, like this:

backend:
  image: scalable-container-image
  build:
    context: ./../
    dockerfile: ./docker/Dockerfile
    target: production
  expose:
    - "3000"
  networks:
    - mainnet
  deploy:
    replicas: 2
backend-cron-manager:
  container_name: cron-manager
  image: scalable-container-image
  build:
    context: ./../
    dockerfile: ./docker/Dockerfile
    target: production
  environment:
    INSTANCE_ID: instance1
  expose:
    - "3000"
  networks:
    - mainnet

Now you can use, for example, Nginx as a load balancer between the two replicas and the cron manager. Remember that all of them are actually the same app!

And now in your code:

import { Cron, CronExpression } from '@nestjs/schedule';

export class DeleteFilesCronService {
  @Cron(CronExpression.EVERY_4_HOURS)
  private async handleDeleteFiles() {
    if (process.env.INSTANCE_ID) {
      // handle delete files
    }
  }
}

Olpe

Answer 4

There are two practical approachs that I would pick:

• If you use k8s, then it support CronJob out of the box: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ First, expose the cronjob as an endpoint and configure it so it only allow network from internal k8s cluster (you don't want someone from the internet trigger it). k8s let you schedule to run a container, in this case, you can use https://hub.docker.com/r/curlimages/curl to make a request to the endpoint. The code would look like this:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: http-call-job
spec:
  schedule: "0 * * * *"  # Runs at the beginning of every hour
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: http-caller
            image: curlimages/curl:latest
            args:
            - /bin/sh
            - -c
            - "curl -X GET http://example.com/endpoint"
          restartPolicy: OnFailure
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 1
  # this will make sure only one Job run is executed at a time
  concurrencyPolicy: Forbid

• If you don't use k8s, then make your own locking mechanism. Yes, it's not that hard, it actually pretty easy. Just create table, open a transaction to verify that you're the first one before creating a new record.

CREATE TABLE CronJobRuns (
   JobName VARCHAR(255)
   StarteddAt DateTime
);

Certification