How to debug issues with Amazon SQS subscription to SNS

Question

I want to build a pub/sub messaging system into my services that are hosted on Amazon Web Services, and creating SQS queues that subscribe to SNS topics seems like the obvious direction to take, but I can't get it working at all.

So far my code looks for the topics and the queues at startup and creates anything that's missing. This seems to work, I can see the SNS topic and the SQS queues in the AWS management console, and I can see that the queue is subscribed to the topic, but when I publish messages to the topic nothing ends up in the queue.

Even if I manually publish a message using the 'Publish' button in the management console the queue is still empty.

I changed the permissions on both the topic and the queue to 'everyone can do everything' just to eliminate this possibility. My code receives a message ID in response to the publish and there are no errors, every API call returns a 200 (OK) status.

Where can I go from here to figure out why it's not working?

Fipple

By a process of trial end error I figured out that SNS is not compatible with FIFO queues, but it's only mentioned in this tutorial docs.aws.amazon.com/AWSSimpleQueueService/latest/… If anyone knows of a way to debug in this situation I would still like to see your answer because I plan to use SNS/SQS together in the future. – Fipple

2021: Now AWS supports SNS FIFO queues that can integrate with SQS FIFO queues (and only with SNS FIFO queues, no other subscriber types). – Boatsman

Answer 1

A couple of options -

1. Enable CloudTrail and monitor the logs
2. View the CloudWatch logs to identify any permissions issues
3. Open a ticket with AWS support.

Ideally, you wouldn't be creating the resources in your application but instead decouple those into CloudFormation or at a minimum CLI scripts. If you require the ability to dynamically create these resources, using the AWS IoT Message Broker may be a better option since it supports ephemeral messaging resources - http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html

Blacktop

Thank you for your helpful suggestions. These resources need to be created by the application because they are dynamically created and deleted as the result of user actions within the application. – Fipple

Edited my answer to add detail for ephemeral messaging resources. – Blacktop

Answer 2

The SNS --> SQS link has a few gotchas:

• "Amazon SNS isn't currently compatible with FIFO queues." per the note on their Subscribing an Amazon SQS Queue to an Amazon SNS Topic Tutorial
• You have to fiddle with the IAM permissions (see the page on Sending Amazon SNS Messages to Amazon SQS Queues)
• You can't send messages to encrypted queues (see their Server-Side Encryption page)

It would definitely have been easier to figure this out if all this info were consolidated into a single page. The killer for me was #3 - perhaps one of these will be the solution to your issue.

Frannie

#3 was also a killer for me, thanks for your help :) – Aphoristic

#3 is possible now, you can send SNS messages to a SSE SQS Queue. This helped me do it, forums.aws.amazon.com/thread.jspa?threadID=288198 – Hawkeyed

The link for #2 is out of date. I think this is the best current resource: docs.aws.amazon.com/sns/latest/dg/… – Hymnal

Answer 3

I had a similar issue with SQS subscriptions. It turned out that if I create the subscription from the SQS editor it works, but if I create it from the SNS creation screen it accepts the message but never forwards it to the queue.

Pibgorn

Thanks! This solved my issue and makes me think it's some magic (permissions?) AWS is doing for us but differently... – Infusive

Answer 4

To get more detailed information about specific failures you can turn on SNS "Delivery status logging".

Osuna

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Leshalesher