How to send multiple batch messages from a list to an SQS queue using Boto3 and Python

Question

I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.

I can do this one message at a time with the send_message() command, and the code for that is below.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    for i in integerList:
        response = queue.send_message(MessageBody=str(i),
                                      MessageGroupId='TESTING')

However, I'd like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages() command, but I'm not sure how to build the Entries= attribute for the batch send. I'm breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)], but the next steps are unclear.

Gnarly

Answer 1

Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    maxBatchSize = 10 #current maximum allowed
    chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
    for chunk in chunks:
        entries = []
        for x in chunk:
            entry = {'Id': str(x), 
                     'MessageBody': str(x), 
                     'MessageGroupId': 'ANYTHINGYOUWANT'}
            entries.append(entry)
        response = queue.send_messages(Entries=entries)

Gnarly

Why would you use the send_message() method for batches instead of send_message_batch()? – Inhibition

@Inhibition send_message_batch is only available on the client, the example has send_messages (note the plural) which is using the SQS service resource. – Stark

Answer 2

According to the documentation Entries is a list of messages.
For each entry in Entries the parameters type are detailed at the link.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    entries = []

    for i in integerList:
        entry =  {
            'Id': 'id%s' % str(integerList[i]),
            'MessageBody': str(integerList[i])
            }
        entries.append(entry)

    response = queue.send_messages(entries)

Adjutant

The maximum allowed individual message size and the maximum total payload size (the sum of the individual lengths of all of the batched messages) are both 256 KB; You cannot send more then 256KB – Adjutant

You can only send 10 messages in a single batch, but you can send more than 1 batch. I want to send n messages (1,000,000) in n/10 batches where each batch is less than 256KB – Gnarly

When I try this I get: 'sqs.ServiceResource' object has no attribute 'send_messages' (whereas the 'send_message' works fine). I just upgraded to the latest boto3 1.12.3 and still getting this error. – Dealings

I put an example on GitHub that shows this with a little more context. – Lezlielg

Answer 3

According to the new version of boto3, the method to use is send_message_batch, here is the updated code:

import boto3

sqs_client = boto3.client("sqs", region_name="eu-west-1")

entries = []
for i in integer_list:
    entry =  {
        'Id': 'id%s' % str(integerList[i]),
        'MessageBody': str(integerList[i])
    }
    entries.append(entry)
    

response = queue.send_message_batch(Entries=entries, QueueUrl=<your_sqs_url>)

Source: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.send_message_batch

Serotine

Incorrect, both send_messages and send_message_batch still exist in the the new boto3 version – Whodunit