AWS SNS — How generic should topics be and when should we reuse/create topics?

Question

We are introducing SNS + SQS to handle event production and propagation in our micro services architecture, which has so far relied on HTTPS calls to communicate with each other. We are considering connecting multiple SQS queues onto one SNS topic. The events in the queues will then be consumed by a lambda or a service running in EC2.

My question is, how generic should the topics be? When should we create new topics?

Say, we have a user domain which needs to publish two events—created and deleted. Two options we are considering are:

OPTION A: Have two topics, "user-created" and "user-deleted". Each topic guarantees a single event type.

• the consumers would not have to worry about discarding events that they are not interested in, as they know already know the messages coming from a "user-created" topic is only related to user creations.
• multiple different parts of the code publishing to the same topic

OPTION B: Have one topic, "users", that accepts multiple event types

• the consumers would have an additional responsibility of filtering through the events or taking different actions depending on the type of the event (they can also configure their queues subscriptions to filter certain event types)

• can ensure a single publisher for each topic

Does anyone have a strong preference for either of the options and why would that be?

On a related note, where would you include the cloud configuration for each of the resources? (should the queue resource creation be deployed together with the consumers, or should they live independently from any of the publishers/consumers?)

Sarre

Answer 1

I think you should go with Option B and keep all events concerning a given "domain" (e.g. "user") in a single topic:

• keeps your infrastructure simple
• you might introduce services interested in multiple event types (e.g. "create" and "delete"). Its kind of tricky to get the ordering right consuming this from two topics; imagine a "user-delete" event arriving before the "user-create" event
• throughput might be an issue, this really depends on your domain (creating and deleting users doesn't sound like a high volume issue)
• think about changes in the data structures in your topics, introducing changes in two or more topics simultaneously can get complicated pretty fast

Concerning your other question: Keep your topic/infrastructure configuration separate from your services. It's an individual piece of infrastructure (like a database) and should kept separate; especially if you introduce more consumers & producers to your system.

EDIT: This might be an example "setup":

• Repository user-service contains the service/lambda code, cloudformation/terraform templates for the service and its topic subscriptions
• Repository sns contains all cloudformation/terraform templates concerning SNS topics
• Repository sqs contains all cloudformation/terraform templates concerning SQS topics

You can think about keeping the SNS & SQS infra code in a single repository (the last two), but I would strongly recommend everything specific to a certain service/lambda to be kept in separate repositories.

Generally it helps to think about your topics as a "database", this line of thinking should point you in the right direction for all your questions.

Mockingbird

Thanks for the detailed response! If I understood correctly your advice regarding separating the infra configuration, do you mean having one, say, serverless.yml which contains the config for the SNS topic, it's subscriptions, and the SQS queues (+ storing etc) and a consumer lambda would be deployed separately? And does that mean you put all event related infra in one repo regardless of domain, or one infra repo for domain=topic? – Sarre

Answer 2

There is also this idea of MessageAttributes when you publish to SNS/SQS.
So if you decide to go with OPTION B, you can send the details of what operation it is as a message attribute. Now you can have say a Lambda subscribe to all events via SQS, or at the point of creating SNS->SQS subscription you can configure message filtering to deliver only CREATE events to the SQS.
If you decide to include it in a message body, you can setup filters on this too.
This gives you options to do both - have event processors accepting ALL or just SPECIFIC events.

https://docs.aws.amazon.com/sns/latest/dg/example-filter-policies.html

Inoperable