Push vs Pull Queues on Google App Engine

Question

So far, no one (not even the GAE docs) has been able to give me a really clear description of what the difference is between a push queue and a pull queue.

My understanding is that the Task Queue API allows you to define task queues where work can be enqueued to. Somehow, this works with GAE's auto-scaling feature so that you don't need to explicitly manage the number of worker threads consuming tasks off these queues: GAE just does it for you.

But nowhere can I find a "King's English" description of the difference between push and pull queues. What is a "push queue" pushing? What is a "pull queue" pulling? Are they both configured inside queues.xml?

Gifted

Answer 1

In a pull queue you enqueue tasks into the queue and your code needs to pull them, you pull them by leasing tasks from the queue and deleting the tasks. if you don't delete the tasks and the lease time is expired the system will return the tasks back to the queue.

You can use pull queue (for example) to aggregate a multiple work units that can be processed together. Another example: queuing task that will be pull by an external machine (like EC2 or gCompute) in order to process the task in a manner that AppEngine can't.

In push queue you enqueue tasks into the queue but AppEngine will dequeue them and will run them on the handler specified by the task. You can control the task processing rate, how to control task execution failures and AppEngine will decide how many instances (threads) to use todo the processing.

Lorielorien

Ahhh thanks @Shay Erlichmen (+1) - so to summarize: a "pull" queue requires you to manually dequeue and process tasks explicitly in your code; whereas with a "push" queue you just specify a handler and a config and GAE takes care of the rest. Is this a fair assessment? Also, a few followups: (1) does queues.xml only apply to push queues? (2) I assume the Task Queue API contains code for explicitly dequeuing/deleting task from a pull queue, yes? Thanks again! – Gifted

@pnongrata Your summery is correct, and as for (1) no, you also define pull queues in queues.xml ,at the minimum if the queue isn't in the xml it doesn't exists. and (2) yes, look for lease_queue (dequeue) and delete_tasks. – Lorielorien