What is the difference between Amazon SNS and Amazon SQS?
Asked Answered
D

9

728

When would I use SNS versus SQS, and why are they always coupled together?

Demulsify answered 3/12, 2012 at 10:22 Comment(7)
What I understand from your comment is described in following flow.. is it correct? Publisher --> SNS --> SQL (holding messages in queue) --> Subscriber (currently offline)Amundson
aws.amazon.com/blogs/aws/…Naughty
@Amundson you mean SQS and not SQL?Lashelllasher
wisdomplexus.com/blogs/amazon-sqs-vs-snsOpposable
There are Wikipedia articles for both: Amazon SNS and Amazon SQSJalisajalisco
SQS lacks pub/sub, while SNS lacks competing-consumers: learn.microsoft.com/en-us/azure/architecture/patterns/… and also needs SQS for dead-letter support. In short, you should really be using both together.Guillot
ably.com/topic/aws-sns-vs-sqsListless
F
995

SNS is a distributed publish-subscribe system. Messages are pushed to subscribers as and when they are sent by publishers to SNS.

SQS is distributed queuing system. Messages are not pushed to receivers. Receivers have to poll or pull messages from SQS. Messages can't be received by multiple receivers at the same time. Any one receiver can receive a message, process and delete it. Other receivers do not receive the same message later. Polling inherently introduces some latency in message delivery in SQS unlike SNS where messages are immediately pushed to subscribers. SNS supports several end points such as email, SMS, HTTP end point and SQS. If you want unknown number and type of subscribers to receive messages, you need SNS.

You don't have to couple SNS and SQS always. You can have SNS send messages to email, SMS or HTTP end point apart from SQS. There are advantages to coupling SNS with SQS. You may not want an external service to make connections to your hosts (a firewall may block all incoming connections to your host from outside).

Your end point may just die because of heavy volume of messages. Email and SMS maybe not your choice of processing messages quickly. By coupling SNS with SQS, you can receive messages at your pace. It allows clients to be offline, tolerant to network and host failures. You also achieve guaranteed delivery. If you configure SNS to send messages to an HTTP end point or email or SMS, several failures to send message may result in messages being dropped.

SQS is mainly used to decouple applications or integrate applications. Messages can be stored in SQS for a short duration of time (maximum 14 days). SNS distributes several copies of messages to several subscribers. For example, let’s say you want to replicate data generated by an application to several storage systems. You could use SNS and send this data to multiple subscribers, each replicating the messages it receives to different storage systems (S3, hard disk on your host, database, etc.).

Faldstool answered 3/12, 2012 at 22:14 Comment(11)
so basically, to implement something like push notification messages, it is recommended to use SNS and SQS so the pushes with sns will be queued until the user is only to retrieve them from the queue? Is it possible to create a queue per user?Demulsify
Yeah. You can have as many subscribers as you want for SNS. You can have notifications sent to multiple queues.Faldstool
Hi sorry, i see this question is old but i am wondering about SQS does it know and store offline messages? Because APNS do not store offline messages, only the newest message. Would it know when the IOS devices are offline and stores the offline messages straight away? And send it out later when the devices are back online?Precisian
@Precisian SQS does store messages until someone takes them out of the queue (and delete of course) or the queue retention time expires. Queue retention time can be configured. There is a limit to it, which can be found in documentation. APNS has nothing to do with SQS. SQS is not for push notifications to mobile devices.Faldstool
@NickGinanto Queue per user probably isn't what you want. You'd probably want one queue for each service that then handles user-specific messages. This diagram may help: aws.amazon.com/blogs/aws/…Guyenne
So lets say I am publishing messages to an SNS, and I have some web service running on some host which is supposed to receive the service. If the service fails for whatever reason, when the SNS tries to push messages to the service, will they just be lost? So if I make the SNS push to an SQS, and the service polls the SQS, if the service goes down, then the messages from the SNS are still stored in the SQS, right?Drawers
@Drawers If SNS fails to deliver message, it will retry according to the retry policy configured on the topic. I would look in their documentation on how to setup retry policies. If you make SNS push to SQS, messages will stay in your queue until retention period expires. You can configure retention period on the queue as well. Check their documentation.Faldstool
It should probably be noted that as of mid 2018 SQS can trigger lambdas and therefore is more akin to a pubsub in that case.Haldeman
@Haldeman Agree. See read.acloud.guru/… for referenceLining
This is by far, the best and simple explanation I came across about the two AWS services.Jockstrap
In SQS a message is Pushed into the queue, which is still a single point of failure. Change my mind.Inbound
F
416

Here's a comparison of the two:

Entity Type

  • SQS: Queue (Similar to JMS)
  • SNS: Topic (Pub/Sub system)

Message consumption

  • SQS: Pull Mechanism - Consumers poll and pull messages from SQS
  • SNS: Push Mechanism - SNS Pushes messages to consumers

Use Case

  • SQS: Decoupling two applications and allowing parallel asynchronous processing
  • SNS: Fanout - Processing the same message in multiple ways

Persistence

  • SQS: Messages are persisted for some (configurable) duration if no consumer is available (maximum two weeks), so the consumer does not have to be up when messages are added to queue.
  • SNS: No persistence. Whichever consumer is present at the time of message arrival gets the message and the message is deleted. If no consumers are available then the message is lost after a few retries.

Consumer Type

  • SQS: All the consumers are typically identical and hence process the messages in the exact same way (each message is processed once by one consumer, though in rare cases messages may be resent)
  • SNS: The consumers might process the messages in different ways

Sample applications

  • SQS: Jobs framework: The Jobs are submitted to SQS and the consumers at the other end can process the jobs asynchronously. If the job frequency increases, the number of consumers can simply be increased to achieve better throughput.
  • SNS: Image processing. If someone uploads an image to S3 then watermark that image, create a thumbnail and also send a Thank You email. In that case S3 can publish notifications to an SNS topic with three consumers listening to it. The first one watermarks the image, the second one creates a thumbnail and the third one sends a Thank You email. All of them receive the same message (image URL) and do their processing in parallel.
Freebooter answered 14/6, 2018 at 6:49 Comment(9)
if no consumers are available then there is a retry mechanism, even default value is 10 retries.Doti
Nice detailed post. We have different messages for different consumers - What should we do - use SNS and define different topics or use SQS and define different queues? A topic/queue could though have one or more consumers.Histology
If your requirement is that your tooic/queue should have more than one consumers I assume you are saying that the same message should be broadcasted to multiple consumers... And if this assumption of mine is correct then using SNS is the only option available for youFreebooter
I do not think "SQS : All the consumers are supposed to be identical and hence process the messages in exact same way" this is right. I have used SQS where two different AWS services are picking up from SQS queue and processing the message in their own way (different application logic in those different services). Am I missing something?Emarginate
@Emarginate I will need to understand your use case but to me it makes no sense that 2 SQS consumers are processing messages in non-identical ways. That is a use case for SNSFreebooter
What puts that restriction of being identical on SQS consumers? You can very well have different consumers for a single message. One consumer can process it (say email) one way and not remove it from the SQS, another consumer processes differently (store in DB), while another can do a third thing (confirm email was sent and the message is stored in DB). Just because SNS is documented to be more suitable for 'fan-out' types of tasks doesn't mean the SQS consumers would have that restriction.Plunkett
Yes, but then who deletes the message. Wouldnt the consumer 1 and connsumer 2 and consumer 3 all keep getting the same message again and againFreebooter
medium.com/awesome-cloud/… This article seems to have similar content to this answer plus some additional details.Cetacean
Thanks for pointing that out... But that medium.com article was published on May 25 2019 and this answer was given almost a year earlier on June 14 2018Freebooter
M
71

You can see SNS as a traditional topic which you can have multiple Subscribers. You can have heterogeneous subscribers for one given SNS topic, including Lambda and SQS, for example. You can also send SMS messages or even e-mails out of the box using SNS. One thing to consider in SNS is only one message (notification) is received at once, so you cannot take advantage from batching.

SQS, on the other hand, is nothing but a queue, where you store messages and subscribe one consumer (yes, you can have N consumers to one SQS queue, but it would get messy very quickly and way harder to manage considering all consumers would need to read the message at least once, so one is better off with SNS combined with SQS for this use case, where SNS would push notifications to N SQS queues and every queue would have one subscriber, only) to process these messages. As of Jun 28, 2018, AWS Supports Lambda Triggers for SQS, meaning you don't have to poll for messages any more.

Furthermore, you can configure a DLQ on your source SQS queue to send messages to in case of failure. In case of success, messages are automatically deleted (this is another great improvement), so you don't have to worry about the already processed messages being read again in case you forgot to delete them manually. I suggest taking a look at Lambda Retry Behaviour to better understand how it works.

One great benefit of using SQS is that it enables batch processing. Each batch can contain up to 10 messages, so if 100 messages arrive at once in your SQS queue, then 10 Lambda functions will spin up (considering the default auto-scaling behaviour for Lambda) and they'll process these 100 messages (keep in mind this is the happy path as in practice, a few more Lambda functions could spin up reading less than the 10 messages in the batch, but you get the idea). If you posted these same 100 messages to SNS, however, 100 Lambda functions would spin up, unnecessarily increasing costs and using up your Lambda concurrency.

However, if you are still running traditional servers (like EC2 instances), you will still need to poll for messages and manage them manually.

You also have FIFO SQS queues, which guarantee the delivery order of the messages. SQS FIFO is also supported as an event source for Lambda as of November 2019

Even though there's some overlap in their use cases, both SQS and SNS have their own spotlight.

Use SNS if:

  • multiple subscribers is a requirement
  • sending SMS/E-mail out of the box is handy

Use SQS if:

  • only one subscriber is needed
  • batching is important
Monicamonie answered 20/3, 2019 at 13:24 Comment(2)
Distributing tasks among many consumers is a primary intended use case for SQS. (Regardless of whether consumers retrieve 1 message or 10 at a time, it is up to the application to control how many consumers exist concurrently.) It differs from SNS because SNS tries to send every message to all subscribers, whereas SQS tries to partition the messages among the consumers.Everyday
Max batch size is now 10k for Standard Queues; still 10 for FIFO Queues.Evangelical
E
67

AWS SNS is a publisher subscriber network, where subscribers can subscribe to topics and will receive messages whenever a publisher publishes to that topic.

AWS SQS is a queue service, which stores messages in a queue. SQS cannot deliver any messages, where an external service (lambda, EC2, etc.) is needed to poll SQS and grab messages from SQS.

SNS and SQS can be used together for multiple reasons.

  1. There may be different kinds of subscribers where some need the immediate delivery of messages, where some would require the message to persist, for later usage via polling. See this link.

  2. The "Fanout Pattern." This is for the asynchronous processing of messages. When a message is published to SNS, it can distribute it to multiple SQS queues in parallel. This can be great when loading thumbnails in an application in parallel, when images are being published. See this link.

  3. Persistent storage. When a service that is going to process a message is not reliable. In a case like this, if SNS pushes a notification to a Service, and that service is unavailable, then the notification will be lost. Therefore we can use SQS as a persistent storage and then process it afterwards.

Evictee answered 18/4, 2018 at 5:36 Comment(3)
Your comment is the only one clearly explaining how SNS and SQS can be combined. Thanks!Gabby
I am a newbie to AWS, but has 3. changed since the answer was written? [docs.aws.amazon.com/sns/latest/dg/… Delivery Retries) Amazon SNS defines a delivery policy for each delivery protocol. The delivery policy defines how Amazon SNS retries the delivery of messages when server-side errors occur (when the system that hosts the subscribed endpoint becomes unavailable). When the delivery policy is exhausted, Amazon SNS stops retrying the delivery and discards the message—unless a dead-letter queue is attached to the subscription.Shippy
@Shippy 3. hasn't changed; it refers to the consumer of the SQS becoming unavailable. That documentation you cite describes a new feature of SNS in which failure to deliver a message to the consumer(s) (in this case, the SQS) after exhausting your retries results in SNS putting that message put onto a dedicated SQS queue for to handle transmission error. This means instead of the message being lost entirely, you have it saved and can re-attempt transmission through the SNS topic at a later date.Tetrasyllable
K
37

From the AWS documentation:

Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates.

Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available.

Fanout to Amazon SQS queues

Kerch answered 12/3, 2014 at 4:31 Comment(1)
SQS can "poll waiting for a message to come in" (which is similar to a push). But it only processes that message once so it's not a broadcast push, FWIW :)Hypothyroidism
H
21

Following are the major differences between the main messaging technologies on AWS (SQS, SNS, +EventBridge). In order to choose a particular AWS service, we should know the functionalities a service provides as well as its comparison with other services.

The below diagram summarizes the main similarities as well as differences between this service.

enter image description here

Hyracoid answered 5/4, 2021 at 5:0 Comment(0)
E
13

In simple terms,

  • SNS - sends messages to the subscriber using push mechanism and no need of pull.

  • SQS - it is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components.

A common pattern is to use SNS to publish messages to Amazon SQS queues to reliably send messages to one or many system components asynchronously.

Reference from Amazon SNS FAQs.

Erdman answered 27/5, 2019 at 20:13 Comment(5)
SQS cannot send a message to many systems as it does not fan out the messages. Yes, many pollers can pull messages from it, but if one the consumers deletes the message then other subscribers won't be able to consume the same message again. SNS is preferred over SQS if you want to achieve a fan out pattern. Also, if the visibilityTimeout is set, than no other systems will be able to consume the message once it's being processed by other system.Monicamonie
A common pattern is to use SNS to publish messages to Amazon SQS queues to reliably send messages to one or many system components asynchronously. Reference from aws.amazon.com/sns/faqsErdman
If that's what you meant (SNS -> multiple SQS queues) then please edit your answer and I will happily remove my downvote. The way you put it, it sounds like SQS can fan out.Monicamonie
Yes.. thats where the confusion was. I have edited it .. Thanks :)Erdman
This is a start, but the question was about the differences, etc.: "When would I use SNS versus SQS, and why are they always coupled together?" Perhaps address that more directly? You can edit your answer (without "Edit:", "Update:", or similar - the answer should appear as if it was written today).Jalisajalisco
L
5

There are some key distinctions between SNS and SQS:

  • SNS supports A2A and A2P communication, while SQS supports only A2A communication.

  • SNS is a pub/sub system, while SQS is a queuing system. You'd typically use SNS to send the same message to multiple consumers via topics. In comparison, in most scenarios, each message in an SQS queue is processed by only one consumer. With SQS, messages are delivered through a long polling (pull) mechanism, while SNS uses a push mechanism to immediately deliver messages to subscribed endpoints.

  • SNS is typically used for applications that need real time notifications, while SQS is more suited for message processing use cases.

  • SNS does not persist messages - it delivers them to subscribers that are present, and then deletes them. In comparison, SQS can persist messages (from 1 minute to 14 days).

Individually, Amazon SQS and SNS are used for different use cases. You can, however, use them together in some scenarios.

Listless answered 22/12, 2022 at 15:7 Comment(0)
E
1

One reason for coupling SQS and SNS would be for data processing pipelines.

Let's say you are generating three kinds of product, and that products B & C are both derived from the same intermediate product A. For each kind of product (i.e., for each segment of the pipeline) you set up:

  • a compute resource (maybe a lambda function, or a cluster of virtual machines, or an autoscaling kubernetes job) to generate the product.
  • a queue (describing units of work that need to be performed) to partition the work across the compute resource (so that each unit of work is processed exactly once, but separate units of work can be processed separately in parallel and asynchronously with each other).
  • a news feed (announcing outputs that have been produced).

Then arrange so that the input queues for B & C are both subscribing to the output announcements of A.

This makes the pipeline modular on the level of infrastructure. Rather than having a monolithic server application that generates all three products together, different stages of the pipeline can utilise different hardware resources (for example, perhaps stage B is very memory intensive, but the two other stages can be performed with cheaper hardware/services). This also makes it easier to iterate on the development of one pipeline segment without disrupting delivery of the other products.

Everyday answered 22/12, 2021 at 9:36 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.