Understanding AWS FIFO Queue behaviour

Question

Was playing around with AWS SQS FIFO Queue locally in localstack with AWS Java sdk v2 & Spring Boot.

I created one endpoint to send messages through one publisher and three endpoints to receive/poll messages from queue via three consumers in Spring boot controller classes.

I created the FIFO queue with following properties -

RECEIVE_MESSAGE_WAIT_TIME_SECONDS = 20 seconds (long poll)
VISIBILITY_TIMEOUT = 60 seconds
FIFO_QUEUE = true
CONTENT_BASED_DEDUPLICATION = true

Each consumer could fetch at max 3 messages (At least 1 if available, up-to 3) per each poll request.

I published 5 messages to the queue (in order). They are -

Message Group Id | Deduplication Id
-----------------------------------
        A        |       A1
        A        |       A2
        A        |       A3
        A        |       A4
        A        |       A5

From log -

2022-06-01 16:13:26.474  INFO 27918 --- [nio-9099-exec-1] c.p.sqs.service.SqsPublisherServiceImpl  : sendMsgRequest SendMessageRequest(QueueUrl=http://localhost:4566/000000000000/dev-priyam-fifo-queue.fifo, MessageBody={"id":"A1"}, MessageDeduplicationId=A1, MessageGroupId=A)
2022-06-01 16:13:26.600  INFO 27918 --- [nio-9099-exec-2] c.p.sqs.service.SqsPublisherServiceImpl  : sendMsgRequest SendMessageRequest(QueueUrl=http://localhost:4566/000000000000/dev-priyam-fifo-queue.fifo, MessageBody={"id":"A2"}, MessageDeduplicationId=A2, MessageGroupId=A)
2022-06-01 16:13:26.700  INFO 27918 --- [nio-9099-exec-3] c.p.sqs.service.SqsPublisherServiceImpl  : sendMsgRequest SendMessageRequest(QueueUrl=http://localhost:4566/000000000000/dev-priyam-fifo-queue.fifo, MessageBody={"id":"A3"}, MessageDeduplicationId=A3, MessageGroupId=A)
2022-06-01 16:13:26.785  INFO 27918 --- [nio-9099-exec-4] c.p.sqs.service.SqsPublisherServiceImpl  : sendMsgRequest SendMessageRequest(QueueUrl=http://localhost:4566/000000000000/dev-priyam-fifo-queue.fifo, MessageBody={"id":"A4"}, MessageDeduplicationId=A4, MessageGroupId=A)
2022-06-01 16:13:26.843  INFO 27918 --- [nio-9099-exec-5] c.p.sqs.service.SqsPublisherServiceImpl  : sendMsgRequest SendMessageRequest(QueueUrl=http://localhost:4566/000000000000/dev-priyam-fifo-queue.fifo, MessageBody={"id":"A5"}, MessageDeduplicationId=A5, MessageGroupId=A)

I then started polling from consumers randomly. My observation is stated below -

1. A1, A2 and A3 were polled. They were polled but not deleted (intentionally). So they went back to the queue after visibility timeout (60 seconds) was over.
2. In the next poll, A3 and A4 were polled. Again, they were polled but not deleted. So they went back to the queue after 60 seconds.
3. In the next poll, A4 and A5 were polled. Again, they were polled but not deleted. So they went back to the queue after 60 seconds.
4. In the next poll (and all following polls) A5 was polled. And I kept getting only A5 from here on.

Now I want to understand why I am getting this behaviour. The whole selling point of FIFO is getting ordered messages (per same message group id). My expectation after step 1 was, I will get one of A1, A1 A2 or A1 A2 A3 in the next poll (step 2) - but this didn't happen.

Can anyone explain what is happening here?

My github repo : https://github.com/tahniat-ashraf/java-aws-sqs-101

Sounder

Is the group id same for all your messages? – Dejected

Yes. The group id is same. Deduplication id is different for all. – Sounder

You must be doing something wrong in your test. As you said, the entire point of FIFO is to have order. Maybe your messages get removed from the queue, as maybe you have setup only 1 attempt for delivery. – Dejected

Yeah I guess so. But if there was only 1 attempt for delivery, multiple attempts for the last one from each batch (A3, A4, A5 in separate cases) wouldn't happen. I'm missing something else. – Sounder

You would have to describe all steps and settings necessary to reproduce your experiment. Its difficult to speculate without knowing what exactly did you do. – Dejected

Oh, you are using localstack. Did you try to run it on AWS? It could be bug in localstack. – Dejected

Indeed it was a bug of localstack as explained by chris – Sounder

Answer 1

I believe this is a known issue in localstack, when using both CONTENT_BASED_DEDUPLICATION=true and providing a MessageDeduplicationId.

SQS supports either content-based duplication, or manual deduplication via a deduplication ID. It does not support both.

Try running this on an actual SQS queue - or change your configuration as described in the localstack issue.

Denature