Using Amazon SQS with multiple consumers

Question

I have a service-based application that uses Amazon SQS with multiple queues and multiple consumers. I am doing this so that I can implement an event-based architecture and decouple all the services, where the different services react to changes in state of other systems. For example:

• Registration Service:
  • Emits event 'registration-new' when a new user registers.
• User Service:
  • Emits event 'user-updated' when user is updated.
• Search Service:
  • Reads from queue 'registration-new' and indexes user in search.
  • Reads from queue 'user-updated' and updates user in search.
• Metrics Service:
  • Reads from 'registration-new' queue and sends to Mixpanel.
  • Reads from queue 'user-updated' and sends to Mixpanel.

I'm having a number of issues:

• A message can be received multiple times when doing polling. I can design a lot of the systems to be idempotent, but for some services (such as the metrics service) that would be much more difficult.
• A message needs to be manually deleted from the queue in SQS. I have thought of implementing a "message-handling-service" that handles the deletion of messages when all the services have received them (each service would emit a 'message-acknowledged' event after handling a message).

I guess my question is this: what patterns should I use to ensure that I can have multiple consumers for a single queue in SQS, while ensuring that the messages also get delivered and deleted reliably. Thank you for your help.

Beilul

Answer 1

I think you are doing it wrong.

It looks to me like you are using the same queue to do multiple different things. You are better of using a single queue for a single purpose.

Instead of putting an event into the 'registration-new' queue and then having two different services poll that queue, and BOTH needing to read that message and both doing something different with it (and then needing a 3rd process that is supposed to delete that message after the other 2 have processed it).

One queue should be used for one purpose.

• Create a 'index-user-search' queue and a 'send to mixpanels' queue, so the search service reads from the search queues, indexes the user and immediately deletes the message.

• The mixpanel-service reads from the mix-panels queue, processes the
  message and deletes the message.

The registration service, instead of emiting a 'registration-new' to a single queue, now emits it to two queues.

To take it one step better, add SNS into the mix here and have the registration service emit an SNS message to the 'registration-new' topic (not queue), and then subscribe both of the queues I mentioned above, to that topic in a 'fan-out' pattern.

https://aws.amazon.com/blogs/aws/queues-and-notifications-now-best-friends/

Both queues will receive the message, but you only load it into SNS once - if down the road a 3rd unrelated service needs to also process 'registration-new' events, you create another queue and subscribe it to the topic as well - it can run with no dependencies or knowledge of what the other services are doing - that is the goal.

Minuteman

This is perfect. I didn't know you could do that with SNS! Thank you! – Beilul

What about FIFO qeue? Can you implement strict ordering with SNS? – Kinaesthesia

@ÖzerS. currently the fan-out pattern described above using SNS & SQS supports only SQS standard queues. As of today, SQS FIFO queues are not supported. I haven't yet found out if there is any intention from AWS to support this in future. – Microcrystalline

I've read that AWS Kineses could be an interesting option to solve the "multiple consumer" design, but I know nothing about it. I don't like having to add one more layer (and paid service) just to forward a message. I'd also have to find a way to programmatically create the SNS topics through my CI/CD process, so it's versioned and doesn't rely on human steps. – Kherson

Found this for FIFO cases docs.aws.amazon.com/sns/latest/dg/… – Charlottcharlotta

Sounds like you might want a Step Function to orchestrate the message between all these components. – Adame

Answer 2

The primary use-case for multiple consumers of a queue is scaling-out.

The mechanism that allows for multiple consumers is the Visibility Timeout, which gives a consumer time to process and delete a message without it being consumed concurrently by another consumer.

To address the "At-Least-Once Delivery" property of Standard Queues, the consuming service should be idempotent. If that isn't possible, one possible solution is to use FIFO queues, but this mode has a limited message delivery rate and is not compatible with SNS subscription.

Oracular

* FIFO queues are now compatible with SNS – Quintan

*idempotent (sp) – Oracular

SNS and SQS now have a compatible FIFO mode. FYI, it looks like it will require a .fifo extension for both the topic and the queue (which will be reflected in the ARNs as well). – Oracular

Answer 3

They even have a tutorial on how to create a fanout scenario using the combo SNS+SQS.

https://aws.amazon.com/getting-started/tutorials/send-fanout-event-notifications/

Too bad it does not support FIFO queues so you have to be careful to handle out of order messages.

It would be nice if they had a consistent hashing solution to have multiple competing consumers while respecting the message order.

Daggna