AWS Api Gateway connect to SQS with message attributes

Question

I have connected an Api Gateway method to push onto a SQS Queue by following this tutorial.

https://dzone.com/articles/creating-aws-service-proxy-for-amazon-sqs

This all works fine but I would like to add some Message Attributes to my messages using some path parameters but I can not get it to work.

This is the current Mapping Template from the tutorial.

Action=SendMessage&MessageBody=$util.urlEncode($util.escapeJavaScript($input.json('$')))

I've tried to keep the 'application/x-www-form-urlencoded' format and add the MessageAttributes to this by changing it to (using a static value for now)

    Action=SendMessage&MessageBody=$util.urlEncode($util.escapeJavaScript($input.json('$')))
&MessageAttribute[0][Name]=foo&MessageAttribute[0][Type]=String&MessageAttribute[0][Value]=bar

but I get this error back

{
  "Error": {
    "Code": "MalformedQueryString",
    "Message": "Keys may not contain [",
    "Type": "Sender"
  },
  "RequestId": "ea121e6e-ca37-5d14-b92b-4a2c6fedf403"
}

How should I encode an array of MessageAttribues without using '['

Irrelevant

Answer 1

I was trying to pass SQS MessageAttributes in from AWS API gateway using a Mapping Template. I ended up tracing the equivalent request from the aws cli to submit a message to SQS with MessageAttributes. It involved setting up a local https proxy and looking at the raw http POST request that the aws cli generated. From the request I was able to put together the working mapping template:

Action=SendMessage&MessageBody=message+with+attribute&MessageAttribute.1.Name=somename&MessageAttribute.1.Value.DataType=String&MessageAttribute.1.Value.StringValue=somevalue

This was the original aws cli command I used for testing:

aws sqs send-message --queue-url "queue/url"  --message-body "message with attribute" --message-attributes '{"somename" : { "DataType":"String", "StringValue":"somevalue"}}' --no-verify-ssl

The aws docs also show the format https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html#API_SendMessage_Examples

Sample Request

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue/
?Action=SendMessage
&MessageBody=This+is+a+test+message
&MessageAttribute.1.Name=my_attribute_name_1
&MessageAttribute.1.Value.StringValue=my_attribute_value_1
&MessageAttribute.1.Value.DataType=String
&MessageAttribute.2.Name=my_attribute_name_2
&MessageAttribute.2.Value.StringValue=my_attribute_value_2
&MessageAttribute.2.Value.DataType=String
&Expires=2020-05-05T22%3A52%3A43PST
&Version=2012-11-05
&AUTHPARAMS

Note:

• it's MessageAttribute not MessageAttributes
• to pass a map parameter you don't need .member.

Mini

Answer 2

The SQS API Reference doesn't make this entirely clear and, in fact, seems to be somewhat inconsistent with what the service really expects to see. On the wire, it uses the .member.N. notation, and N is 1-based rather than 0-based. Also, MessageAttributes is plural.

The expected format looks like this:

&MessageAttributes.member.1.Name=source_ip##
&MessageAttributes.member.1.Value.DataType=String##
&MessageAttributes.member.1.Value.StringValue=$util.urlEncode($context.identity.sourceIp)##
&MessageAttributes.member.2.Name=user_agent##
&MessageAttributes.member.2.Value.DataType=String##
&MessageAttributes.member.2.Value.StringValue=$util.urlEncode($context.identity.userAgent)##
&MessageAttributes.member.3.Name=stage##
&MessageAttributes.member.3.Value.DataType=String##
&MessageAttributes.member.3.Value.StringValue=$util.urlEncode($context.stage)##

The ## are for readability. You can't have newlines in a mapping template when you're building a web form, because they would end up in the form itself, and ## tells VTL that everything from here to the end of the line is to be disregarded, including the newline at the end of the line. Embedded newlines would otherwise make the form construction incorrect.

Note also that the API does not allow these values to be blank, so if there is a chance that any attribute won't be present, you'll need extra logic to test this and replace the attribute with something else, because the index numbers (iirc) must be consecutive.

Why we're using a body mapping template to build a web form for API Gateway + SQS, and more information on how to configure it, is also explained in more detail in my answer to Is it possible to POST to the SQS URL using the POST body?

Obstetrics

Thanks for the quick response but I'm getting the exact error you described with attributes being blank using your example code. Endpoint request body after transformations: Action=SendMessage&MessageBody=%7B%7D&MessageAttributes.member.1.Name=source_ip&MessageAttributes.member.1.Value.DataType=String&MessageAttributes.member.1.Value.StringValue=test-invoke-source-ip (just one values for comment character limit) – Irrelevant

It isn't clear from what you posted why this would be an error, since attribute 1 clearly isn't blank... but you can cheat with some quotation marks. It isn't proper quoting but will prevent an uncaptured value from being blank in the form. Compare the following with what's in the answer above, noting the addition of two ", and see if you can use this to at least get a successful response -- &MessageAttributes.member.1.Value.StringValue="$util.urlEncode($context.identity.sourceIp)"## – Obstetrics

Yes it clearly isn't blank but I'm getting the same error. Even with a hardcoded value Action=SendMessage&MessageBody=%7B%7D&MessageAttributes.member.1.Name=source_ip&MessageAttributes.member.1.Value.DataType=String&MessageAttributes.member.1.Value.StringValue="foo" – Irrelevant

{ "Error": { "Code": "InvalidParameterValue", "Message": "The request must contain non-empty message attribute name.", "Type": "Sender" }, "RequestId": "ded496eb-afc7-5c56-89af-c2a860080ea5" } – Irrelevant

It appears to be complaining of an empty attribute name (not an empty value). Strange. – Obstetrics

I get the same error as @JohnWebb. I removed the "member" notation and it worked fine. &MessageAttributes.1.Name=source_ip## instead of &MessageAttributes.member.1.Name=source_ip## – Willodeanwilloughby

Answer 3

In my case I had to do API Gateway ---> SNS and use MessageAttributes for filtering. It's a similar process. After following Michael - sqlbot solution in this post and other here Is it possible to POST to the SQS url using the post body I'm sharing my Mapping Template and POST body payload that worked for me.

Mapping Template:

Action=Publish##
&TopicArn=$util.urlEncode('arn:aws:sns:us-west-2:111111:TOPIC_NAME')##
&Message=$util.urlEncode($input.json('$.Message'))##
&Subject=$util.urlEncode($input.json('$.Subject'))##
&MessageAttributes.member.1.Name=hostName##
&MessageAttributes.member.1.Value.DataType=String##
&MessageAttributes.member.1.Value.StringValue=$util.escapeJavaScript($input.json('$.HostNameValue'))##

Note that

&MessageAttributes.member.1.Name=hostName##

Is hardcoded since it's a KEY and if you mapping it form POST payload you'll get an error Message attribute name invalid (it takes it as is with "").

This one

&MessageAttributes.member.1.Value.StringValue=$util.escapeJavaScript($input.json('$.HostNameValue'))##

Will send HostNameValue as is as well. Since we use urlEncode it's gonna look like this when it hits SNS and therefore your filter will not work.

  "MessageAttributes": {
    "hostName": {
      "Type": "String",
      "Value": "\"your.host.name\""
    }

POST body payload:

{
    "Subject": "YOLO",
    "Message": "Bla",
    "HostNameValue": "your.host.name"
}

Holotype

Hi Dionis, I am using the below template ``` Action=Publish &TopicArn=$util.urlEncode("arn:aws:sns:eu-west-2:036220108529:booking-topic-imod-4225") &&Message=$util.urlEncode($input.json('$')) ``` But I am getting the error Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter" – Comedy

Answer 4

Michael's answer nearly did it for me. But I was running into this error :

The request must contain non-empty message (user) attribute names

To get around this, you just need to delete the property member that was included in his solution.

Stroganoff