Google App Engine Task Deadline

Question

is there a way to handle any soft deadline while executing a task ? The DeadlineExceededError is thrown after 10 minutes of execution and I am given few seconds after that to do some stuff. I want to clean up somethings before the task dies and create a new task. This may take few seconds. Is there a way to do this by capturing any exception like around 9 mins. I know I can manually throw an exception after 9 mins. But can this be done automatically by GAE ?

class FillMtxHandler():

def post(self,index,user,seqlen):

    try :           
        FillMtx(index,user,seqlen)

    except DeadlineExceededError:

        deferred.defer(self.post,index,user,seqlen)

The above is my code. index is a list and starts from 0. It will be incremented inside FillMtx. Once a deadline exceeded error is thrown, i want to continue from where index was incremented last. I get the following error

   The API call taskqueue.BulkAdd() was explicitly cancelled.
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 517, in __call__
    handler.post(*groups)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 258, in post
    run(self.request.body)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 124, in run
    return func(*args, **kwds)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 166, in invoke_member
    return getattr(obj, membername)(*args, **kwargs)
  File "/base/data/home/apps/0xxopdp/3.347813391084738922/fillmtx.py", line 204, in post
    deferred.defer(self.post,index,user,seqlen)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 241, in defer
    return task.add(queue, transactional=transactional)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/taskqueue/taskqueue.py", line 688, in add
    return Queue(queue_name).add(self, transactional=transactional)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/taskqueue/taskqueue.py", line 744, in add
    self.__AddTasks(tasks, transactional)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/taskqueue/taskqueue.py", line 770, in __AddTasks
    apiproxy_stub_map.MakeSyncCall('taskqueue', 'BulkAdd', request, response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 86, in MakeSyncCall
    return stubmap.MakeSyncCall(service, call, request, response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 286, in MakeSyncCall
    rpc.CheckSuccess()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 126, in CheckSuccess
    raise self.exception
CancelledError: The API call taskqueue.BulkAdd() was explicitly cancelled.

I find that a new task has been created and queued up. But why does GAE throw this error still?

Holystone

Answer 1

You can't control when you get the soft deadline exceeded error. Instead, you should use your own timer (take the wallclock time when you start, and compare it to the current time on each trip around your main loop), and abort when you're near enough the limit that you want to stop.

Pearman

Thanks for the reply. In your post about deferred, you add a new task after catching a DeadlineExceededError. Can we be sure that the task will be definitely added to the queue before the hard deadline error ? – Holystone

@Holystone It should generally be added, but it's possible that it won't be, as you've observed. If it doesn't, though, your task will be requeued (since it didn't exit cleanly); your code can then check if it's already done the work, and skip straight to the cleanup phase if it did. – Pearman

Answer 2

You don't have to raise an exception after 9 minutes; when the soft deadline exception raises, you have enough time to add a clean-up task to the Task Queue via deferred.

from google.appengine.ext import deferred
...
try:
  # Do your stuff
except DeadlineExceededError:
  deferred.defer(do_your_cleanup, ..) 

In this way you have 10 minutes to do any clean-up stuff you need in your app.

Kirkpatrick

how can i be sure that the task will be added to the task queue before the hard limit is hit ? i get the "CancelledError: The API call taskqueue.BulkAdd() was explicitly cancelled" even though the task has been added – Holystone

If you run out of grace time the CancelledError kicks in. – Kirkpatrick

deferred.defer and taskqueue.add behave the same way here. In both cases, i get the "Cancelled Error" because the process of adding the task to the queue was not completed before the hard time limit. code.google.com/appengine/articles/deferred.html how is he able to update the db and create a task before the grace period ? – Holystone

I have modified the question. i have added my code and error log – Holystone