Why do we need a queue when using webhooks?

Question

Can anyone clarify what is the purpose of using queue ?

What i understand is that a webhook is just a URL , you do a POST request to that URL and then do some stuff based on the body/data of the request. So why i need to queue the data and store it in a database then loop through the database again and perform the stuff.

Guessrope

Answer 1

The short answer is, you don't have to use a queue. A webhook is just an HTTP request (typically POST) notifying your application of some type of event. The reason you might want to consider a queue is because of typical issues you could run into.

One of these is because of response time back to the webhook requester (source). Many sources want a response (HTTP status 200) as quickly as possible so they can dequeue the request from their webhook system. If processing the webhook takes some time, a source will typically advise you to use a queue to defer the lengthier process asynchronous to the 200 response to the webhook.

Another possible reason could be for removing duplicate requests. There is no guarantee with webhooks that you will only receive a single request per event. A queue can be used to de-dupe these requests.

I would recommend you stick with a simple request handler if possible, then evolve a more sophisticated handler if you run into issues. Consider queues as a potential design approach if you run into issues like those above.

Milicent

Accurate answer but I'll add another considerations. Typically when on the receiving end you have no control over the rate the webhooks are received. In many cases, this can cause issues where volume increase unexpectedly causing performance degradation and outages. Your own webhook handling method might be limited by downstream services like a 3rd party API with rate-limiting you are calling as part of the webhook handler. Now, time for a plug: while you ideally want to differ the processing it doesn't have to be complex to set up. We've built hookdeck.com to handle it all for you. – Antisepticize

Answer 2

You need some way to prevent a conflict if the webhook is invoked multiple times very close together.

It doesn't necessarily have to be a queue, though. If the webhook performs database queries and updates, you can use a transaction to ensure that this is atomic for each invocation.

In this respect, it's little different from any other web utility. You should do something similar in scripts that process web forms.

Emeldaemelen

Answer 3

Generally, webhooks are used with messaging queues like Rabbitmq and Kafka. This is used when your application heavily relies on webhook for critical things like syncing live data etc, with the help of messaging queue you can be sure when servers are down or your system is not able to process high traffic queue can handle webhook on your behalf and make sure no webhook data is loss. So if your application doesn't require webhook for critical things you can go with the URL method else you should use Queue.

Outwear