AWS gives us Amazon MQ but how can I trigger a Lambda?

Question

Superb news about the Amazon MQ service but now the question arises on how I can trigger a Lambda function (Node.js) on a message on a specific Queue?

I was thinking if I somehow can get a SNS topic posted on a message PUT or some other trigger that can fire a Lambda to consume the message from the Queue...

Any suggestions?

Peltast

Answer 1

AWS recently announced

https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambda-supports-amazon-mq-apache-activemq-event-source/

We can now add Trigger as MQ in lambda. Then Configure

• Broker url
• Batch size
• Queue Name

Hatten

Yes, I noticed that a few weeks back but we've built everything on top of SQS and EventBridge now... :) – Peltast

Answer 2

There isn't a native way to do this. Amazon's managed ActiveMQ service is simply a managed deployment of ActiveMQ running in EC2. It has no integration with other services.

You'd need to write a queue consumer and have it running on a server and listening to the queue on ActiveMQ and publishing those messages to SNS or invoking the Lambda function directly via the Lambda API, etc.

(At least for now.)

Oliva

Thanks m8! I had guessed as much... Too bad really as it would have been great to have a Lambda trigger. As SQS also lacks one it's a bit frustrating. Could ActiveMQ "call" an API Gateway API? Now we can place API GW in a VPC so it would be "internal"... – Peltast

@Peltast the "internal" aspect of API Gateway refers to it being able to make connections into resources inside your VPC (a web server) to submit HTTP requests without coming in via the Internet. ActiveMQ doesn't reach out -- just like SQS, the consumers connect to it, first, then it starts spewing messages. One notable difference is that the consumers can use a persistent TCP connection, rather than HTTP long polling. You'd need a homebrew bridge to interconnect with other things. – Oliva

I've been tinkering with an SQS/Lambda bridge that I've considered offering as SaaS product, where you could configure it to do things like "take messages from the queue and use them to invoke Lambda functions, with a maximum concurrency of 5," etc. but I have hesitated to try marketing it because I assume AWS will introduce something like that at some point... but they didn't announce anything like that at re:Invent last week. Let me know if you might be interested in beta testing it. – Oliva

Sure, yes, we'd be interested in getting more details! We were initially using SQS to trigger platform events when we had an ESB product but we migrated to Lambda and I built a "SQS Poller" in Lambda that was invoking itself every 5 minutes to keep it running but eventually it went down anyway... If you have you can send any details of what you have come up with to me! Thanks! – Peltast

Answer 3

Here is one approach that AWS describes - https://aws.amazon.com/blogs/compute/invoking-aws-lambda-from-amazon-mq/

Basically, have a cloud watch trigger for the lambda, start polling for MQ messages and process those messages

Youngling

That article is flawed. It opens with with a graphic and the explanation of the cloud watch trigger method. The example then goes on to demonstrate one lambda on a 5 minute wake up timer that pulls from the queue – Gentilesse

Here is a repo that gives a more complete example than the blog post github.com/aws-samples/amazonmq-invoke-aws-lambda – Gentilesse