Efficient way to check whether SQS queue is empty

Question

I have a SQS Queue from which messages are read by multiple hosts. I want to run some job (business logic) after all the messages in the queue have been processed.

How can I check that the queue is empty?

Yes, I can check for ApproximateNumberOfMessages and ApproximateNumberOfMessagesNotVisible queue attributes but these are approximate numbers. I want to stop my hosts polling for messages in the queue when there are no messages left and then run the required job.

Any ideas? Thanks

Bunnie

Answer 1

You could simply note empty receives from the API response while you're polling. Concerning CloudWatch, there is another metric that would be a better fit for this. From the documentation:

NumberOfEmptyReceives

The number of ReceiveMessage API calls that did not return a message.

Units: Count

Valid Statistics: Average, Minimum, Maximum, Sum, Data Samples (displays as Sample Count in the Amazon SQS console)

Some additional info:

• This metric only gets populated every 5 minutes. If you set up an alarm based on this metric, this means your minimum period should be 5 minutes.
• Sum is the most sensible statistic for your use case. If NumberOfEmptyReceives > 0, your polling job checked the queue and received no messages.

I personally used this metric to set up a cloudwatch alarm that will scale down an autoscaling group that hosts my polling job after the sum of NumberOfEmptyReceives > 0 for several consecutive periods. I like doing consecutive periods because it makes it more evident that the queue was not only empty, but has stayed empty.

Parfitt

Hi, I am now using a combination of output of ReceiveMessageRequest, ApproximateNumberOfMessages and ApproximateNumberOfMessagesNotVisible along with Long Polling to determine if the queue is empty or not. If the result of ReceiveMessageRequest is null && ApproximateNumberOfMessages == 0 && ApproximateNumberOfMessagesNotVisible==0, then can I be sure that there would be no message left in the queue? – Bunnie

Think so. ApproximateNumberOfMessagesNotVisible probably isn't extremely useful for this; there could be messages in-flight at the same instance CloudWatch samples and give you a false positive for queue depth. For the cloudwatch metrics, make sure you're sampling over several periods to verify that the queue is actually empty, and that there wasn't just a fluke or CloudWatch service error. – Parfitt

Answer 2

You could trigger your post business logic on the cloud watch metric related to queue depth. When the depth is 0 then you can send a SNS notification or start a lambda function.

Also this cloud watch metric is better then others since this is actual message count reported by sqs service.

Hitt

Answer 3

In case anyone else sees this: AWS documentation recommends checking that the number of messages in a queue is zero for several minutes straight before considering that queue empty. This is due to the distributed nature of SQS and because these metrics are approximate.

To confirm that a queue is empty (AWS CLI, AWS API)

1. Stop all producers from sending messages.

2. Repeatedly run one of the following commands:

   • AWS CLI: get-queue-attributes
   • AWS API: GetQueueAttributes

3. Observe the metrics for the following attributes:

   • ApproximateNumberOfMessagesDelayed
   • ApproximateNumberOfMessagesNotVisible
   • ApproximateNumberOfMessages

   When all of them are 0 for several minutes, the queue is empty.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html

You can have your hosts check the number of messages in the queue every few seconds for several minutes and keep track of how long the queue has been continuously empty. Once you confirm the queue is empty, you can run the required job.

Alternatively, if you are using a FIFO queue with a single message group and know beforehand which message from the producer will be the last one, you can add a message attribute that marks that message as the last one.

Slap