AWS SQS to receive message from outside of AWS

Question

my company has a messaging system which sends real-time messages in JSON format, and it's not built on AWS, and will not have any VPN connection with AWS.

our team is trying to use AWS SQS to receive these messages, which will then have DynamoDB process JSON messages to TSV, then load into RDS.

however, as per the FAQ, SQS can only receive message from within AWS. https://aws.amazon.com/sqs/faqs/

Q: Who can perform operations on a message queue?

Only an AWS account owner (or an AWS account that the account owner has delegated rights to can perform operations on an Amazon SQS message queue.

In order to use SQS, one way I can think of is to create a public-facing EC2 instance, which receives messages and passes over to SQS.

My questions here are:

1. is my idea correct?
2. if it's correct, can you share any details on how to build any applications on this EC2 instance to achieve the functionality (I have no experience on application development, your insights are really appreciated!)
3. is there any easier/better options in AWS that can achieve the goal to receive message in my use case?

Accipiter

Answer 1

1. is my idea correct?

No, it isn't.

You're misinterpreting the (admittedly somewhat unclear) information in the FAQ.

SQS is accessible and usable from anywhere on the Internet. Its only exposed interface is HTTP(S). In fact, from inside EC2, SQS is not accessible unless the EC2 instance actually has outbound access to the Internet.

The point being made in the documentation is not that you need to be "inside" AWS to use queues, but rather that you need to be in possession of an authorized set of AWS credentials in order to work with queues.¹

If you have an AWS account, you have credentials, and you can use SQS. There is no requirement that you access the queue from "inside" AWS.

Choose the endpoint closest to your servers (for lowest latency) and you should find it open and accessible, from anywhere.

---

¹Queues can be configured to allow anonymous acccess after they are created. (Don't do it, I'm just saying it is possible.) This section of the FAQ seems to be referring to a subset of operations, such as creating queues.

Jones

Hi Michael, thanks for the reply. I should've made it clearer in my description. To my understanding, if the company's queue messaging system has AWS credential or if it's on EC2 with a role having permission to access SQS, it can send messages to SQS. However, what I got stuck of is that the company's queue messaging system is generic, which does't use any AWS credential when broadcasting messages. I also tried POSTMAN to send message to the SQS endpoint without AWS credential, which failed too. So I wonder if any workaround for this use case. Please correct me if I'm wrong, thank you. – Accipiter

That is a different issue. The (potentially) even bigger problem than credentials is that you have to speak in the format expected by the SQS API. A reasonably straightforward workaround in your case -- I suspect -- would be AWS API Gateway, which can proxy and rewrite incoming HTTPS requests for AWS services like SQS. Is that what you want? Accept messages over HTTPS and drop them into a queue? – Jones

Hi Michael, I took a look at AWS API Gateway, which seems like a good fit. Yes, accepting messages over HTTPS and dropping them into SQS is what I want. Would you please advise any development work is need to make it work? Thank you very much. – Accipiter

Just configuration, I believe... request & response integration mappings. – Jones

The end goal is to have a real-time (or as close as possible to real-time) message accepting and converting service. For example, having a service accept the messages from the company's messaging system, and having another service (or probably the same service) convert the received JSON-format messages into TSV-format, and load it into RDS. All in real-time rather than batch processing. – Accipiter

Answer 2

You can access sqs from anywhere once you have proper permission through accesskey&secret key or IAM role.

SQS is not specific to vpc

Ullyot

Answer 3

I was not able to write to SQS from an external service. I found some partial explanations but got stuck at the role creation.

The alternative I found is using AWS services Lambda + API Gateway to write to SQS.

This tutorial was extremely helpful, explaining all the steps in great details: https://startupnextdoor.com/adding-to-sqs-queue-using-aws-lambda-and-a-serverless-api-endpoint/

Valora

Answer 4

It is clear that you try to do this :

Take message from your company messaging system, send it to SQS.

It is not wrong using your method (using EC2 as a bridge). However, you don't need EC2 to connect to SQS.

All AWS services can be access using AWS API(e.g. Python boto3, etc) from internet, as long as you provide the correct credential. So you can put your "middleware" in anywhere as long as you are able establish connection to the said services.

So there is lots of more options available to you. e.g. trigger from your messaging system; use AWS Lambda, etc.

Licko

Answer 5

Thanks for sharing the information and your insights with me!

I have tested below solution, which works for my use case:

1. created an endpoint in AWS API Gateway, which is able to receive messages from company messaging system, a system that does not carry AWS credentials

2. created a Lambda function triggered by API Gateway, so once a message arrives, Lambda will digest the JSON message and convert it to TSV, and then load into RDS

Accipiter