SQS Message always stays inflight

Question

I have the following code retrieving messages from a SQS queue. I am using a AmazonSQSBufferedAsyncClient to retrieve the message from Queue. A fixed delay SingleThreadedExecutor wakes up every 5 mins calling receiveMessage. Long polling is enabled in the Queue

@Service
public class AmazonQueueService
    implements QueueService<String> {

    @Autowired
    private AmazonSQSBufferedAsyncClient sqsAsyncClient;

    @Value("${aws.sqs.queueUrl}")
    private String queueUrl;

    @Override
    public List<Message<String>> receiveMessage() {
        ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl);
        ReceiveMessageResult result = sqsAsyncClient.receiveMessage(receiveMessageRequest);

        LOG.debug("Size=" + result.getMessages().size());

        return Lists.transform(result.getMessages(), ......);
    }

    .....
}

The problem is when I check AWS console, the message is always in-flight, but it is never received in the application (Size is always printed as 0) . Looks like the AmazonSQSBufferedAsyncClient is reading the message from queue but not returning in the receiveMessage call.

Any ideas?

Cincture

How is the AmazonSQSBufferedAsyncClient configured (timeouts, threads, etc)? And how is the queue configured, regarding the visibility timeout of the messages? Or are you specifying a visibility timeout when sending the messages? – Tele

Something that just occured: you seem to be using some kind of dependency injection (@Autowired). Could you possibly be creating multiple instances of AmazonSQSBufferedAsyncClient instead of a single instance, a singleton, that is shared among all the instances of your class AmazonQueueService? If this is the case, they would be competing for the messages. You could try printing the .hashCode() of the AmazonSQSBufferedAsyncClient just to ensure that you have a single instance... – Tele

Im prettry sure there is only a single instance of `AmazonSQSBufferedAsyncClient'. The queue attributes are as below – Cincture

Im prettry sure there is only a single instance of `AmazonSQSBufferedAsyncClient'. The queue attributes are as below Visibility Timeout: 2 minutes; Retention Period: 4 days; Receive Message Wait Time: 20 seconds. While making the request no attributes are set, so it should use the queue attributes by default. – Cincture

Answer 1

Finally figured this out. The problem manifests with the combination of queue visibility timeout (2 mins) and the scheduledExecutor delay (5 mins).

Increasing the visibility timeout to 15 mins solved the problem.

My theory is -> The AmazonSQSBufferedAsyncClient retrieves message and keeps it in the buffer waiting for receiveMessage call. As the executor delay is 5 mins, the visibility of the message times out befor receiveMessage is called and the message is returned to the queue. It also looks like the message is picked from the queue almost immediately. And now for whatever reason a call to receiveMessage does not receive the message. Increasing the timeout which gave a chance for the receiveMessage call to happen before a timeout event, solved the problem, I guess.

Any other possible explanation?

Cincture

Answer 2

For me problem was my messages were being read by lambda even before I see them on console. So the result was I was not able to poll for the messages from sqs console.

Squirearchy

Oh my god thank you so much, I was working on a LOCAL worker and didn't realize I was going autopilot and had created a Lambda actually consuming the SQS messages. Jesus, I was so bewildered. – Fiesole

Answer 3

You have to delete the message from the queue when you're done with it. If you don't, it's going to stay in-flight until it times out and then go right back to the queue. It is designed this way so you will never lose messages. If your program crashes before it finishes handling the message and deleting it, the message will go right back to the queue.

From the basic Java example (SampleDriver.java):

QMessage message = messages.get(0);

System.out.println("\nMessage received");
System.out.println("  message id:     " + message.getId());
System.out.println("  receipt handle: " + message.getReceiptHandle());
System.out.println(" message content: " + message.getContent());

testQueue.deleteMessage(message.getReceiptHandle()); // <===== here

Kaja

Yes. But deleting comes after receiving and processing the message at least once, I dont receive the message from sqsAsyncClient.receiveMessage call – Cincture

Ah. Well, if it's in-flight it's something on the client side. Either you have another client getting the message and not deleting it or something is wrong the Java call. Have you tried other libraries like Python boto? – Kaja

Answer 4

In my case, I was using short polling, and sometimes it was receiving more than 1 message but I was processing only one. Thus was losing track of messages. Since the message was never processed its visibility timeout was never reached and thus message appeared in inflight messages.

Television