Lambda SQS Trigger Batch window and Batch size not working as expected

Question

I have an AWS SQS (Standard Queue) which listens to third party SNS. I have a lambda setup which has SQS trigger with Batch size 10000 and Batch window 300. My SQS receives approx. 150 messages at a time but lambda gets triggered in batches of 20-30 messages at a time even i configured Batch size 10000. I don't understand why this is happening... even the SQS have enough messages and enough time (300 seconds Batch window) to fill the batch, its not doing it.

I googled for the issue and found that maximum payload size for lambda can be 6MB. I checked my message and its approx. 2.5 KB per message. so 30*2.5 = 75 KB only and not touching the limit 6MB.

Additionally, I suspected lambda concurrency so i have set it up to value 1 only. so no parallel lambda instances.

Can somebody help me to understand where the problem is please?

Manslaughter

Answer 1

Lambda uses five parallel long-polling connections to check your queue. So if you have 150 msgs, each connection gets about 30 msgs, exactly explaining what you see.

Sadly, you can't change the number of these connections. There are always five.

Speight

Thanks for pointing out the reason Marcin. I was really struggling to find that out. Now I know the reason, I will find out alternative/work-around for this. is there anything you can suggest regarding work-around/alternative to this issue? – Manslaughter

@JaydipKalkani No problem. Sadly I don't know what you can do, except setting up your own solution and not using SQS->Lambda integration provided by AWS. – Speight

Thank you Marcin. I realized that even after 5 parallel long polling, concurrency can be handled by us. I have set concurrency to 1 so no parallel lambda function execution will happen and that will help me to achieve what i wanted. Sadly 2 problems are there, #1 more latency in output due to no parallel execution and #2 unnecessary lambda executions (5 instead of only 1) due to 5 parallel lambda connections. Thank you again :) – Manslaughter