How to implement a priority queue using SQS(Amazon simple queue service)

Question

I have a situation when a msg fails and I would like to replay that msg with the highest priority using python boto package so he will be taken first. If I'm not wrong SQS queue does not support priority queue, so I would like to implement something simple.

Important note: when a msg fails I no longer have the message object, I only persist the receipt_handle so I can delete the message(if there was more than x retries) / change timeout visibility in order to push him back to queue.

Thanks!

Brotherton

Answer 1

I don't think there is any way to do this with a single SQS queue. You have no control over delivery of messages and, therefore, no way to impose a priority on messages. If you find a way, I would love to hear about it.

I think you could possibly use two queues (or more generally N queues where N is the number of levels of priority) but even this seems impossible if you don't actually have the message object at the time you determine that it has failed. You would need the message object so that the data could be written to the high-priority queue.

I'm not sure this actually qualifies as an answer 8^)

Bocock

Yup, there is no way to do it with single SQS queue. SQS assumes unordered items and they don't provide FIFO guarantees. – Jacksnipe

SQS now provides FIFO capabilities but you cannot set a message to jump in front of the queue, it is strictly First In First Out. docs.aws.amazon.com/AWSSimpleQueueService/latest/… – Targett

Answer 2

As far as I know AWS SQS does not provide a native way or doing a priority queue (single queue priority). If you are open to considering other options, RabbitMQ can do this. Client can specify a priority of 0-255 in the message, and the queue will prioritize a higher priority message gets to the customer first.

For more information, please look at https://www.rabbitmq.com/priority.html

Terrorism

Answer 3

By "when a msg fails", if you meant "processing failure" then you could look into Dead Letter Queue (DLQ) feature that comes with SQS. You can set the receive count threshold to move the failed messages to DLQ. Each DLQ is associated with an SQS.

In your case, you could make "max receive count" = 1 and you deal with that message seperately

Thiamine

Answer 4

Looking at the original question, we need to have a prioritized retry queue which means:

• the items in the queue should be prioritized because low-priority items may depend on high-priority items, e.g. the low-priority item can be successfully processed only after a high-priority item successfully processed;
• we need to implement multiple retries for queue items;

in this case yes, we can use SQS with some not too important assumptions.

1. Let's say your message is in JSON format and it contains a Priority field. Add it to your SQS.
2. At the consumer, get the message, process it, and if it's failed, set visibility timeout to (priority)*(60minutes) queue
3. Setup maxReceiveCount for the queue equal to the number of retries you need or configure the retention period to move the message to the dead letter queue after a specific number of retries.

So, in theory, the low-priority request may be executed before high-priority only once at the beginning, and then it will be "moved after" high-priority messages using visibility timeout.

Ceiba