How to add SQS message attributes in SNS subscription?

Question

The documentation for AWS SNS and SQS have sections about message attributes. But there is no explanation how to have SQS message attributes when that queue is subscribed to a SNS topic.

Is there a way to configure AWS SNS to add particular message attributes to the SQS messages send via the subscription?

Doan

When an Amazon SQS queue is subscribed to an Amazon SNS topic, the SQS message will contain the message sent to (and forwarded from) SNS. What particular of attributes did you want? – Sweeten

"SQS Message Attributes" is a thing. Separate from the message body. – Doan

That is correct. However, when SNS sends a message to SQS, it populates the SQS message body. What attributes from SNS were you hoping to have populated as an SQS message attribute? – Sweeten

I want to have certain pre-defined attributes on the SQS messages generated from this SNS topic. For example messages on one topic would add an attribute path: foo, another SNS topic sending to the same SQS queue would add an attribute path: bar. According to docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html there seems to be a reference, but nothing else is documented. *Important* To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True. – Doan

See this Question in 2024: How to set up AWS SNS for message filtering? – Sweeten

Answer 1

Note: See other answer for better response, using Raw Message Delivery

---

While the Using Amazon SNS Message Attributes documentation sending Amazon SNS message attributes to Amazon SQS, it appears that the attributes are sent in the body of the message rather than being attached as message attributes on the resulting Amazon SQS messages.

For example, I did the following:

• Created an Amazon SNS topic
• Created an Amazon SQS queue and subscribed it to the SNS topic
• Published a message to SNS

I published via the AWS Command-Line Interface (CLI):

aws sns publish --topic-arn arn:aws:sns:ap-southeast-2:123456789012:foo --message msg --subject subj --message-attributes '{"somename" : { "DataType":"String", "StringValue":"somevalue"}}'

(I got syntax help from map datatype in aws cli)

The resulting message in SQS showed the attributes as part of the message:

{
  "Type" : "Notification",
  "MessageId" : "53e3adad-723a-5eae-a7b7-fc0468ec2d37",
  "TopicArn" : "arn:aws:sns:ap-southeast-2:123456789012:foo",
  "Subject" : "subj",
  "Message" : "msg",
  "Timestamp" : "2017-05-29T12:48:22.186Z",
  ...
  "MessageAttributes" : {
    "somename" : {"Type":"String","Value":"somevalue"}
  }
}

It would be much nicer if these attributes were attached to the SQS message as official SQS attributes. Alas, it seems this is not the case.

Sweeten

I assume from your answer, that at the moment there is no way to make SNS actually create SQS attributes in any way. And most probably the only way to add SQS attributes is to use one of the SDKs or the API directly without a service like SNS. – Doan

@Evgeny that's correct. Unfortunately, the SNS/SQS integration does not handle Message Attributes in the intuitive way. You could also, for example, use SNS > Lambda > SQS, with custom code in the Lambda function to populate the SQS Message Attributes from the SNS Message Attributes. – Edaedacious

@Michael-sqlbot Already working on that. Appears that CloudWatch Events SQS target also does not support SQS message attributes, so working on that as well :( – Doan

Did you forget to change the subscription attribute RawMessageDelivery, which is false by default? Example using aws cli: aws sns set-subscription-attributes --subscription-arn 'subscriptionarn' --attribute-name RawMessageDelivery --attribute-value true – Sanctus

This is not the correct answer. @BorysGeneralov got it right in his answer: https://mcmap.net/q/467137/-how-to-add-sqs-message-attributes-in-sns-subscription – Genniegennifer

Not sure about the way and the specifics, if you're looking for a quick solution, I had to use this object signature (network is related to my project's business rule): ` MessageAttributes: { network: { DataType: 'String', StringValue: messageBody.network, }, }, ` – Originally

Answer 2

From aws documentation:

To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True. For more information about raw message delivery, see Appendix: Large Payload and Raw Message Delivery. https://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html https://docs.aws.amazon.com/sns/latest/dg/large-payload-raw-message.html

Added an example from real-life project. Hopefully it helps to clarify things. The message published to the sns topic is following:

aws sns publish --topic-arn arn:aws:sns:us-west-2:xxx:pollution-event --message '{"operatorId":3375001,"eventTypeId":1,"eventLevelId":1,"validFrom":"2018-03-10T09:00:00Z","validTo":"2018-03-11T09:00:00Z"}'  --message-attributes '{"Type" : { "DataType":"String", "StringValue":"Orchestration.Services.Model.Pollution.PollutionMessage"}}'

Enable Raw Delivery is false (default). The message received by sqs contains only content, there are no attributes

{
  "Type": "Notification",
  "MessageId": "78d5bc6f-142c-5060-a75c-ef29b774ec66",
  "TopicArn": "arn:aws:sns:eu-west-2:xxx:pollution-event",
  "Message": "{\"validFrom\": \"2018-03-10T09:00:00Z\",\"validTo\": \"2018-03-11T09:00:00Z\",\"eventLevelId\": 1,\"eventTypeId\": 1,\"operatorId\": 3375001}",
  "Timestamp": "2018-04-17T11:33:44.770Z",
  "SignatureVersion": "1",
  "Signature": "xxx==",
  "SigningCertURL": "https://sns.eu-west-2.amazonaws.com/SimpleNotificationService-xxx.pem",
  "UnsubscribeURL": "https://sns.eu-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-2:xxx",
  "MessageAttributes": {
    "Type": {
      "Type": "String",
      "Value": "Orchestration.Services.Model.Pollution.PollutionMessage"
    },
    "AWS.SNS.MOBILE.MPNS.Type": {
      "Type": "String",
      "Value": "token"
    },
    "AWS.SNS.MOBILE.MPNS.NotificationClass": {
      "Type": "String",
      "Value": "realtime"
    },
    "AWS.SNS.MOBILE.WNS.Type": {
      "Type": "String",
      "Value": "wns/badge"
    }
  }
}

Enable Raw delivery is true. The message contains the message attributes and proper content

Sanctus

Have you tested this? According to John's answer, he writes at the very beginning that message attributes are delivered in the body of the SQS message and not as SQS attributes. – Doan

Yes, we had exactly the same issue: we published the message with an attribute, but sns forwarded it without any attribute. The sqs contained a message with json content and which in turn contained the attribute. Enabling Raw Message Delivery fixed the issue, i.e. sns sends the message as-is and the attributes remain – Sanctus

Did you publish a message with an SNS attribute? Because that would mean that you still did not get any SQS attributes set on that message, you just received the published SNS raw data. Not the same thing. – Doan

Ah sorry for confusion, would you adjust the question to make it more clear? As far as I understood you were sending the sns message with attributes, but these attributes were not sent to the sqs which is default behavior. Enabling Raw Delivery will deliver the message together with its attributes to the subscribed sqs queue. However, it seems like this is not what you are looking for. – Sanctus

This should be the accepted answer. Once you change the raw setting from default, the message attributes show up correctly in SQS. – Genniegennifer

Very weird. Although the attributes appear on the AWS console, they are not received via the SDK. Why could that happen? – Spector

Meh, you have to pass the --message-attribute-names parameter in order to receive the message attributes... – Spector

This is the correct answer. Enabling the Raw message delivery will "pass" the message attributes from SNS to SQS as actual message attributes instead of being part of the body. – Appurtenant

Thanks - I can confirm this still is the case in 2020. While I can't see the explicit reference to the quote in the docs anymore, I can confirm the attribute filtering does not work if raw message delivery is not set. AWS needs to make this explicit in the docs – Runesmith

Great answer. Exactly what I needed for the raw message delivery — I need to add --message-attribute-names=All to aws sqs receive-message call. – Ribwort

Answer 3

If you are here because you have a SQS queue that is subscribed to an SNS topic, you checked that your subscription has set the Raw Message Delivery to True but you still cannot read an attribute on your SQS message:

Make sure that your SQS client is not filtering out message attributes.

The code below will only include myAttribute when receiving messages from the SQS queue:

SQS.receiveMessage({
  QueueUrl: queueUrl,
  VisibilityTimeout: 20,
  WaitTimeSeconds: 10,
  MessageAttributeNames: [
    "myAttribute"
  ],
},...

If you want to read the value of some attribute other than myAttribute you will have to specify it (white list) or replace "myAttribute" with "All" to include all SQS attributes.

SQS.receiveMessage({
  MessageAttributeNames: [
    "myAttribute", "myOtherAttribute"
  ],
},...

Reference: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

Inconsumable

Answer 4

Enabled Raw message delivery type while adding SQS subscription for the topic inside SNS

Penile