How to guarantee azure queue FIFO

Question

I understand that MS Azure Queue service document http://msdn.microsoft.com/en-us/library/windowsazure/dd179363.aspx says first out (FIFO) behavior is not guaranteed.

However, our application is such that ALL the messages have to be read and processed in FIFO order. Could anyone please suggest how to achieve a guaranteed FIFO using Azure Queue Service?

Thank you.

Secondguess

Answer 1

The docs say for Azure Storage queues that:

Messages in Storage queues are typically first-in-first-out, but sometimes they can be out of order; for example, when a message's visibility timeout duration expires (for example, as a result of a client application crashing during processing). When the visibility timeout expires, the message becomes visible again on the queue for another worker to dequeue it. At that point, the newly visible message might be placed in the queue (to be dequeued again) after a message that was originally enqueued after it.

Maybe that is good enough for you? Else use Service bus.

Westwardly

Answer 2

The latest Service Bus release offers reliable messaging queuing: Queues, topics and subscriptions

Burgett

Unfortunately, this seems to be answering a different question to what the OP asked. The Azure Service Bus is a different beast to the Azure Queue Service – Guanidine

Answer 3

Unfortunately, many answers misleads to Service Bus Queues but I assume the question is about Storage Queues from the tags mentioned. In Azure Storage Queues, FIFO is not guranteed, whereas in Service Bus, FIFO message ordering is guaranteed and that too, only with the use of a concept called Sessions.

A simple scenario could be, if any consumer receives a message from the queue, it is not visible to you when you are the second receiver. So you assume the second message you received is actually the first message (Where FIFO failed :P)

Consider using Service Bus if this is not your requirement.

Leyden

Answer 4

Adding to @RichBower answer... check out this... Azure Storage Queues vs. Azure Service Bus Queues

MSDN (link retired) http://msdn.microsoft.com/en-us/library/windowsazure/hh767287.aspx

learn.microsoft.com https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted

Finery

Answer 5

I don't know how fast do you want to process the messages, but if you need to have a real FIFO, don't allow Azure's queue to get more than one message at a time.

Use this at your "program.cs" at the top of the function.

    static void Main()
            {
                var config = new JobHostConfiguration();

                if (config.IsDevelopment)
                {
                    config.UseDevelopmentSettings();
                }
                config.Queues.BatchSize = 1; //Number of messages to dequeue at the same time.
                config.Queues.MaxPollingInterval = TimeSpan.FromMilliseconds(100); //Pooling request to the queue.


                JobHost host = new JobHost(config);

....your initial information...

            // The following code ensures that the WebJob will be running continuously
            host.RunAndBlock();

This will get one message at a time with a wait period of 100 miliseconds.

This is working perfectly with a logger webjob to write to files the traze information.

Guffey

This is not the situration described above. Even if you only have one serial producer, messages can still arrive out of order if a single consumer crashes and starts reading while the failed message is not visible. – Thingumabob

@CodeMonkey, Are you sure that if message is not visible, it is not considered a part of the session? Can you provide any link, that proves the behaviour that you mentioned? – Muckworm

@MichaelFreidgeim I think we are confusing Storage Account Queues and Azure Service Bus Queues here. Storage Account Queues have no order guarantees. Sessions is a feature of Azure Service Bus. – Thingumabob

Answer 6

As mentioned here https://www.jayway.com/2013/12/20/message-ordering-on-windows-azure-service-bus-queues/ ordering is not guaranteed also in service bus, except of using recieve and delete mode which is risky

Vinegary

Answer 7

You just need to follow below steps to ensure Message ordering.:

1) Create a Queue with session enabled=false. 2) While saving message in the queue, provide the session id like below:-

var message = new BrokeredMessage(item);
message.SessionId = "LB";
Console.WriteLine("Response from Central Scoring System : " + item);
client.Send(message);

3) While creating receiver for reviving message:-

queueClient.OnMessage(s =>
{
    var body = s.GetBody<string>();
    var messageId = s.MessageId;
    Console.WriteLine("Message Body:" + body);
    Console.WriteLine("Message Id:" + messageId);
});

4) While having the same session id, it would automatically ensure order and give the ordered message.

Thanks!!

Yip

The question is about Storage queues I guess – Leyden