The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue

Question

I'm using a Microsoft Azure Service Bus queue to process calculations and my program runs fine for a few hours but then I start to get this exception for every message that I process from then on. I have no clue where to start since everything runs fine for the first few hours. My code seems to be accurate as well. This the method where I handle the Azure Service Bus message.

public static async Task processCalculations(BrokeredMessage message)
{
    try
    {
        if (message != null)
        {
            if (connection == null || !connection.IsConnected)
            {
                connection = await ConnectionMultiplexer.ConnectAsync("connection,SyncTimeout=10000,ConnectTimeout=10000");
                //connection = ConnectionMultiplexer.Connect("connection,SyncTimeout=10000,ConnectTimeout=10000");
            }
            cache = connection.GetDatabase();
            string sandpKey = message.Properties["sandp"].ToString();
            string dateKey = message.Properties["date"].ToString();
            string symbolclassKey = message.Properties["symbolclass"].ToString();
            string stockdataKey = message.Properties["stockdata"].ToString();
            string stockcomparedataKey = message.Properties["stockcomparedata"].ToString();
            var sandpTask = cache.GetAsync<List<StockData>>(sandpKey);
            var dateTask = cache.GetAsync<DateTime>(dateKey);
            var symbolinfoTask = cache.GetAsync<SymbolInfo>(symbolclassKey);
            var stockdataTask = cache.GetAsync<List<StockData>>(stockdataKey);
            var stockcomparedataTask = cache.GetAsync<List<StockMarketCompare>>(stockcomparedataKey);
            await Task.WhenAll(sandpTask, dateTask, symbolinfoTask,
                stockdataTask, stockcomparedataTask);
            List<StockData> sandp = sandpTask.Result;
            DateTime date = dateTask.Result;
            SymbolInfo symbolinfo = symbolinfoTask.Result;
            List<StockData> stockdata = stockdataTask.Result;
            List<StockMarketCompare> stockcomparedata = stockcomparedataTask.Result;
            StockRating rating = performCalculations(symbolinfo, date, sandp, stockdata, stockcomparedata);
            if (rating != null)
            {
                saveToTable(rating);
                if (message.LockedUntilUtc.Minute <= 1)
                {
                    await message.RenewLockAsync();
                }
                await message.CompleteAsync(); // getting exception here
            }
            else
            {
                Console.WriteLine("Message " + message.MessageId + " Completed!");
                await message.CompleteAsync();
            }
        }
    }
    catch (TimeoutException time)
    {
        Console.WriteLine(time.Message);
    }
    catch (MessageLockLostException locks)
    {
        Console.WriteLine(locks.Message);
    }
    catch (RedisConnectionException redis)
    {
        Console.WriteLine("Start the redis server service!");
    }
    catch (MessagingCommunicationException communication)
    {
        Console.WriteLine(communication.Message);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        Console.WriteLine(ex.StackTrace);
    }
}

I check the time until the lock expiration and I call lock renew if it needs it - it renews the lock with no errors, but I'm still getting this exception.

timeLeft = message.LockedUntilUtc - DateTime.UtcNow;
if (timeLeft.TotalMinutes <= 2)
{
    //Console.WriteLine("Renewed lock! " + ((TimeSpan)(message.LockedUntilUtc - DateTime.UtcNow)).TotalMinutes);
    message.RenewLock();
}

catch (MessageLockLostException locks)
{
    Console.WriteLine("Delivery Count: " + message.DeliveryCount);
    Console.WriteLine("Enqueued Time: " + message.EnqueuedTimeUtc);
    Console.WriteLine("Expires Time: " + message.ExpiresAtUtc);
    Console.WriteLine("Locked Until Time: " + message.LockedUntilUtc);
    Console.WriteLine("Scheduled Enqueue Time: " + message.ScheduledEnqueueTimeUtc);
    Console.WriteLine("Current Time: " + DateTime.UtcNow);
    Console.WriteLine("Time Left: " + timeLeft);
}

All I know so far is that my code runs fine for awhile, and the renew lock gets called and works but I'm still getting the lock exception and inside that exception.

I output the time left and it keeps increasing the time difference as the code runs, which makes me believe that the time until lock expiration is not being changed somehow?

Eldrida

Did you ever resolve this? What was it? – Kee

@FyodorSoikin No I eventually had to give up and work on a different method to do the same thing. From what I could tell, it is a bug in the api but no one from Microsoft ever responded to my posts – Eldrida

By any chance could any of your individual messages take longer than 60 seconds to process? – Linalool

@Linalool I would renew the lock after the intensive calculation was finished and right before it processed the message – Eldrida

@user3610374 have you tried setting the LockDuration to 5 minutes (which is the max setting), the default is 60 seconds and this exception will be thrown if 60 seconds pass before lock renewals. – Linalool

Yes setting it to the max was one of the first things I did – Eldrida

Answer 1

I spent hours trying to understand why I was getting a MessageLockLostException. The reason for me was due to AutoComplete defaulting to true.

If you're going to call message.Complete() (or CompleteAsync()) then you should instantiate an OnMessageOptions object, set AutoComplete to false, and pass it into your OnMessage call.

var options = new OnMessageOptions();
options.AutoComplete = false;

client.OnMessage(processCalculations, options);

Brainwashing

Thanks Jeffrey, this helped; Can you tell why though? – Terrier

@mebjas you should not "complete" a message multiple times. Either let AutoComplete do the work or take care of it yourself (Complete() / CompleteAsync()) but avoid doing both. – Brainwashing

Would setting "autoComplete": false, in the host.json file not work? – Joselynjoseph

Do you know if there's a Python Azure Function equivalent to this? I see from the documentation that I cannot change the AutoComplete to false for 'non-C# functions'. – Aaberg

Answer 2

It took me 2 days to resolve similar issue - same exception.
This exception may have multiple reasons, I'll describe couple of config options that may help you stranger...

ServiceBus Queue or Topic-subscription config:

• Message lock duration on Queue / Topic subscription is too low set it to approx. message processing time

ServiceBusClient options config:

• tryTimeout is too short, set it to ~10s for diagnostics

ServiceBusProcessor options config:

• AutoCompleteMessages is by default set to true, set it to false
• PrefetchCount is too high, for diagnostics set it to 0
• ReceiveMode set it to ServiceBusReceiveMode.PeekLock
• MaxConcurrentCalls for diagnostics set it to 1

After finding correct values (optimized for a given system) I no longer observed any issues.

Madwort

TY - this helped me out massively and i was able to make alot of localhost checks for debugging support. – Infectious

Setting the prefetchCount to 0 resolved my issue. – Nadya

Answer 3

I was having a similar issue. Messages were being handled successfully, but when they went to complete, the Service Bus didn't have a valid lock anymore. It turns out my TopicClient.PrefetchCount was too high.

It appears that the lock begins on all prefetched messages as soon as they are fetched. If your cumulative message processing time surpasses the lock timeout every other prefetched message will fail to complete. It will return to the service bus.

Mastership

What should the prefetch count be? – Match

This is what did it for me! The prefetch count setting is something to consider changing if you are using service bus triggers for long (ish) running jobs. – Effeminacy

@MarcusW hey, I'm facing the same issue, what did you set your prefetch count to? Also, it there a way to clear service bus queue, I'm using the azure web jobs sdk and don't have direct access to queue client. – Formularize

@Formularize I don't really remember but try starting at one and then increasing depending on your performance requirements. Sort of a trial and error game. For monitoring and managing the bus I can recommend this tool: github.com/paolosalvatori/ServiceBusExplorer/releases – Effeminacy

Answer 4

In my case, it was just that I was working on a V2 on my local machine while I had the V1 already deployed up-and-running.

As the V1 was deployed in Azure (closer to the same Queue) and compiled in release mode (versus my local version in debug mode), the deployed version was always "winning" the concurrency for the queue resource.

That's why the message was no longer in the queue: It was consumed by the deployed version of my code. I know it is a little bit dumb.

Wohlert

Using same queue for development and staging creates this issue for me as well :) – Superheterodyne

Answer 5

Instead of renewing the lock manualy, when you create the client subscription, try auto renewing it using the client's OnMessageOptions() like this:

        OnMessageOptions options = new OnMessageOptions();

        options.AutoRenewTimeout = TimeSpan.FromMinutes(1);

        try
        {
            client = Subscription.CreateClient();

            client.OnMessageAsync(MessageReceivedComplete, options);
        }
        catch (Exception ex)
        {
            throw new Exception (ex);
        }

Eidson

What version of ServiceBus are you using? The version I have is v2.2.0.0 and there is not a property call AutoRenewTimeout in the OnMessageOptions class – Vulture

Hi Rogala. I was using version 2.7.0.0. – Eidson