How increase maximum schedule time in Gcloud tasks API?

Question

I have problem with google cloud tasks API. I need create task with far schedule time. Maximum schedule time is 30 days. Is possibility to increase that limit or some problem's bypass?

Mana

Answer 1

Unfortunately, you are unable to increase the quota for the maximum scheduled time. You could get around this by sending your request to an endpoint which would decide to schedule a new task (extend the scheduled date) or execute the task.

The risk of scheduling your tasks very far out is the task must be completed by 31 days. Therefore if you schedule your task for 30 days, it only has 1 day to finish any retries.

Bluet

Thanks. I already came up with the idea extend the scheduled date by recreating tasks. I will implement it. But I don't understand sense of that limit. Can you report it? – Mana

This is a design decision. Google does not charge for the storage space of tasks, so extending that would detrimental to our costs. Would you pay for a higher maximum schedule time? – Bluet

Speaking only for myself, yes. I would. – Commonwealth

Also sharing that I would pay for it. – Onslaught

@AveriKitsch people who need it will currently implement a rescheduling workaround, so the storage increase would happen anyway. – Boron

Answer 2

You cannot create a cloud task where the scheduled time is more than 30 days from now. But you can use a smart workaround, if you feel you really want to use cloud task for your use case. You could consider creating 2 separate queues.

Case 1: Scheduled time is within 30 days - in this case create a task normally in your main queue, and you're good to go.

Case 2: Scheduled time is after 30 days - for example 45 days. In this case you could create a task (in a separate secondary queue) that will run after 30 days, and call an endpoint that will create yet another task in the main queue that will run after 15 days. This way - you can recursively keep creating tasks and eventually execute your task at your desired time. I've tried this approach in one of my past projects and it worked well for me.

Menfolk

Answer 3

If we look at the quotas of cloud tasks, we see that the default maximum schedule time for a task is 30 days as you have said. At the bottom of this page we see that it is possible to request an increase in quota and a link to how to request this. You didn't say what you wanted it increased to but it may be that you can request an increase that will accommodate your needs. It won't do any harm to ask/try.

Later: Sadly the above won't work. Following on from comments, it appears that the maximum schedule time for a task is not an increasable quota. It appears to be fixed at 30 days.

An additional thought is to see if Cloud Scheduler is a more appropriate product for your function/needs. There is a reference to determining when to use one vs the other found here.

Ress

Thanks. Cloud Scheduler looks nice, but I have already implementation with Cloud Tasks. So I prefer to stay with Tasks. I already checked limits settings page (the one you talked about), but I didn't find option to increase "Maximum schedule time". – Mana

I can't use Cloud Scheduler becouse I am passing dynamic data to my tasks. – Mana

Hmmm. I looked again and the docs seems to be clear that only those that are flagged as increasable can be increased and the maximum schedule time is not one of those. Based on this, I'm going to say that the maximum schedule time for a task is fixed to be 30 days and that's the best we can get. – Ress

Assuming we don't find a solution with Cloud Tasks, maybe you can paint a picture of what you are trying to achieve? Obviously, Cloud Tasks is logically ideal but it may be that you could run a custom Job Scheduler (en.wikipedia.org/wiki/List_of_job_scheduler_software) on a compute engine as an alternative to Cloud Tasks. – Ress

Answer 4

As you can see in the quotas and limits official documentation for Cloud Tasks this 30 days limitation and you have nothing to do about it.

For your use case Cloud Scheduler would be more appropriate. If you need to pass dynamic data to your tasks, you can always store that data somewhere, trigger some Cloud Functions to take that data and insert it programatically wherever you want. There are options, but it will not be that easy. Simply, increasing the quota for the Cloud Tasks product is impossible.

Zee

Answer 5

As said by others, Google Cloud Tasks official documentation confirm that the maximum execution time for a task is 30 days. Thus, as explained by @Sagar Jain, for longer tasks you'll have to implement a logic to recursively re-schedule tasks.

Here's how you could do it using a middleware in Express.js:

// Middleware to reschedule Google Cloud Tasks
export const googleTasksScheduleMiddleware = async (req, res, next) => {
  const taskETAHeader = req.headers['google-cloud-tasks-eta'];

  // If no header, skip middleware
  if(taskETAHeader == null || taskETAHeader == ""){
    next()
    return
  }

  const now = Date.now();
  const taskETA = parseInt(taskETAHeader);

  // Time has passed, let's process the task now
  if(taskETA < now) {
    next()
    return
  }

  // It's too early! Reschedule the task
  else {
    // Construct the task.
    createTask(req.method, req.url, req.headers, req.body, taskETA)

    res.send('Re-scheduled')    
    return
  }
}

The trick is in adding an ETA header to your tasks (I'm using Google-Cloud-Tasks-ETA). This way, before executing the task, in your middleware, you can check if the ETA is now (and thus execute the task) or in the future (and thus re-schedule the task).

In fact, doing so let's you create tasks in 1 year (or more) from now. I have talked more deeply about extending Cloud Tasks on DEV.to.

Onslaught