aws FIFO queue returns empty queue even though it has messages available

Question

I have a FIFO queue with over 2 mil messages available. And I want to process them with lambda functions but 9 out of 10 times I poll messages I get a reply that the queue is empty. Which is definitely not true. I tried to change to long poll but it didn't helped. Here is my code for polling messages.

import { create as listenToSqsQueue } from "sqs-consumer";

listenToSqsQueue({
    queueUrl: Config.aws.sqsurl,

    handleMessage: async function (message, done){
        // do some work with `message`s
        Promise.resolve(invokePoller(functionName, message, callback));
        done();
      }
    ,         
    batchSize: 10
}).on("empty", function() {
    callback(undefined, "Queue is empty");
}).on("error", function(error: Error, message: any) {
    console.error(error, message);
    callback(error)
}).start();

Roderick

How many messages are in flight? Typically, people reporting conditions like this have failed to account for SQS imposing strict ordering within each message group. No groups without messages in flight means no messages are available right now, because to deliver a message in thus condition means delivering out of order. – Promethean

@Michael-sqlbot yes you are right, I missed the point that for FIFO queues the messages with the same messageId will be delivered in order. Thanks for your answer – Roderick

Answer 1

I would suggest you to try out a small test as below steps to understand this behavior of FIFO queues yourself.

1. Create a FIFO queue for testing purposes
2. Send two messages to the queue
3. Invoke receive-message API of the queue. You should receive the first message that you entered to the queue
4. Invoke receive-message API multiple times and for subsequent attempts, you get empty responses. After 30 seconds (visibility timeout), you will get the same message, if you invoke the receive-message API. However, you will never receive the second message from the queue.
5. Delete the message that you received in the step 3
6. Invoke receive-message API and you should receive the second message that you sent to the queue

The reason that your queue consumer receives empty responses for receive-message API is the queue consumer has not deleted the received message after processing it. If your FIFO queue delivered other messages while the already delivered message is not deleted, it could violate the guarantee of "First-In-First-Out Delivery". Because the second message could be processed before the first message is processed. This behavior becomes more apparent if you consider a situation where there are multiple queue consumers.

In a summary, I would suggest you to consider deleting a message as a notification to the FIFO queue to instruct that the queue consumer has successfully processed the message and FIFO queue is allowed to deliver next message in the queue. To fix client code, you may modify it to delete the message after it is successfully processed.

Malnutrition

Answer 2

As an alternative solution to the one presented by @Denis Weerasiri, you can set a different Message Group ID for each message, so you can work around the FIFO restriction that applies to messages in the same group.

See Using the Amazon SQS message group ID from the AWS docs.

Actinochemistry

Answer 3

SQS has this caveat where, For FIFO queues, when you receive a message belonging to a particular message group ID, note the following:

1. you must delete or move messages in the current receive call before you can receive more messages from the same group ID. Note: Messages must be moved from inflight available state.

2. You can’t receive messages in other message groups.

More details can be read in this official answer

Tsarina