Connecting http request/response model with asynchronous queue

Question

What's a good way to connect the synchronous http request/response model with an asynchronous queue based model?

When the user's HTTP request comes it generates a work request that goes onto a queue (beanstalkd in this case). One of the workers picks up the request, does the work, and prepares a response.

The queue model is not request/response - there are only requests, not responses. So the question is, how best do we get the response back into the world of HTTP and back to the user?

Ideas:

1. Beanstalkd supports light weight topics or queues (they call them tubes). We could create a tube for each request, have the worker create a message on that tube, and have the http process sit and wait on the tube for the response. Don't particularly like this one since it has apache processes sitting around taking memory.

2. Have the http client poll for the response. The user's initial HTTP request kicks off the job on the queue and returns immediately. The client (the user's browser) polls periodically for a response. On the backend the worker puts its response into memcached, and we connect nginx to memcached so the polling is light weight.

3. Use Comet. Similar to the second option, but with fancier http communication to avoid polling.

I'm leaning towards 2 since it's easy and well know (I haven't used comet yet). I'm guessing there's probably also a much better obvious model I haven't thought of. What do you think?

Teressaterete

I'm having the same problem and in process of evaluating the same options. Can you share what you chosen, how it performed and what are pros and cons of your implemented solution? Thanks – Bucella

Answer 1

Here's how to implement request-response efficiently on JMS which might be helpful (though Java/JMS centric). The general idea is to create a temporary queue per client/thread then use correlationIDs to correlate requests to replies etc.

Castled

Answer 2

Polling is the simple solution; comet is the more efficient solution. You've got it nailed :)

I personally love comet (although I'm biased, since I helped write WebSync), it nicely lets your clients subscribe to a channel and get the message when your server process is ready. Works like a champ.

Westney

Answer 3

I'm looking to implement a Beanstalkd and memcached system to run a number of processes following a request - in this case, looking up information when a user logs in (the number of messages a user has waiting for example). The info is stored in Memcached and then read back on the next page load.

Without knowing a little more about what tasks you are doing though, it's not so easy to say what needs to be done, or how. Option #2 is however the simplest, and that may be all you need - depending on what you are pushing back into the workers.

Foeticide

The tasks are varied and complex - analyzing the incoming data, forming a feature vector based on the data, performing algorithmic matching, and finally spitting out various matches from the classifier. – Teressaterete