Can AWS SQS publish to SNS or is polling SQS required?

Question

We’re presently building an application using AWS and have a need to push msgs into SQS. My question is whether it is possible to have SQS publish a message to an SNS which will trigger a Lambda (susbscribing to the SNS)? The lambda then needs to return an affirmation to the SQS that it received the message, thereby removing that message from SQS.

Is the scenario outlined above possible? Or is the only way to grab a message from SQS, to poll the queue via Lambda, etc?

Thank you in advance for any help provided.

Apologies for misuse of terminology but I’m relatively new to AWS.

Galatians

I have posted a answer which tells now this is possible after 200 reinvent update, if it helped you, do upvote or accept it so that it can help others in the future. – Hautrhin

Answer 1

[ 2022/2023 Update ]

Now this is possible using EventBridgePipes.

I just tried it using SQS as source and SNS as target and it worked.

Hautrhin

Answer 2

SQS cannot publish messages to SNS. SQS can only store the messages. You have to pull the message using SQS Api's.

Hope this helps you!

Adena

Answer 3

SQS does not support push. However, you can setup a lambda function that periodically polls from SQS fairly easy as described here: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/

Shipment

The problem is that "long polling" method is both inefficient and expensive. You lambda will long poll the queue every 10 seconds (by default). That is 267000 requests per month for a single SQS queue triggering a single lambda. I think people are trying to figure out how to get around the limitation. – Arbela

Answer 4

For anyone in the future who stumbles across this, I had to write this script after using SQS as a dead letter for an SNS -> Lambda service. It's not the prettiest code, but it works.

Below is the script but you can also find it here: https://gist.github.com/joshghent/ca4a1272031e2a52af57d5e8ec5d53c5

sqs_to_sns.py

# Usage
# $ python sqs_to_sns.py my-queue-name

import boto3
import sys
import queue
import threading
from datetime import datetime
import json
from uuid import uuid4

work_queue = queue.Queue()

sqs = boto3.resource('sqs')
sns = boto3.client('sns')
sqs_client = boto3.client('sqs')

from_q_name = sys.argv[1]
print(("From: " + from_q_name + " To: SNS"))

from_q = sqs.get_queue_by_name(QueueName=from_q_name)
to_q = sqs.get_queue_by_name(QueueName='backup-queue')

skipped = 0
processed = 0
total = 0


def process_queue():
    while True:
        messages = work_queue.get()

        global total
        total = len(messages)

        for message in messages:
            message_content = json.loads(message.body)

            message.delete()

            print("Backing up Message to dead letter queue - just in case. Id: " +
                  message_content['MessageId'])
            bodies = list()
            bodies.append({'Id': str(uuid4()), 'MessageBody': message.body})

            to_q.send_messages(Entries=bodies)

            response = sns.publish(
                TopicArn=message_content['TopicArn'], Message=message_content['Message'])
            print(("Published Message to Topic " + str(message_content['MessageId']) +
                   ". To TopicArn: " + message_content['TopicArn'] + ". Received response " + json.dumps(response)))

            global processed
            processed = processed + 1


for i in range(10):
    t = threading.Thread(target=process_queue)
    t.daemon = True
    t.start()

while True:
    messages = list()
    for message in from_q.receive_messages(
            MaxNumberOfMessages=10,
            VisibilityTimeout=123,
            WaitTimeSeconds=20):
        messages.append(message)
    work_queue.put(messages)

work_queue.join()
print("Processed " + str(processed) + " of " + str(total) +
      ". Skipped " + str(skipped) + " messages. Exiting")

Lynea