Can I persist SNS (or SQS) messages to S3 through given AWS integrations?

Question

I want to save all SNS messages to S3 so I can keep a complete history. The purpose is to have the ability to go look back when diagnosing and issue, later index and use this to audit, replay, rebuild data, etc. For now, I just want to dump them into S3 (cheap) and later I'll dig into it when needed.

• I don't need to transform or manipulate the data, I can just dump it for now

Can I use AWS provided integrations to accomplish this or do I need to write my own HTTP service or Lambda?

I'd rather not have to write and maintain a lambda or service if I can avoid it.

What I've seen so far is that I can set an HTTP endpoint as an SNS subscriber but it has a different message format than the S3 http api thus requiring a lambda to transform it. Also the initial subscriber confirmation message. Is this not possible and I should get to work on my own service/lambda?

Greenlet

what approach did you go with finally? I have a similar use case where I want to persist messages somewhere on event of subscriber failures. – Procyon

@AndyDufresne I made a lambda that subscribes to the SNS topic and puts an S3 document of every JSON that comes through the SNS topic. Also, added an SQS queue between the two so messages are not lost. This means we have to pay for the lambda to run, which is nothing for low volume but a huge cost across all topics we have. Hopefully one day this integration comes to make this work and cost not necessary! – Greenlet

Answer 1

As far as I know this isn't possible.

It would be pretty trivial to write a lambda which can persist your SNS or SQS messages.

You can also use something like sqs-s3-logger (note this code is based on the fact you couldn't trigger a Lambda from SQS, this is possible now so you don't need a cron job)

Compliant

Thank you! It's very helpful to know it's not possible and save me time! – Greenlet

Answer 2

An SNS topic can be subscribed to by a Kinesis Data Firehose, the Firehose can then publish to an S3 bucket for persistence.

From the documentation:

Trella