ServiceBus RetryExponential Property Meanings

Question

I'm having a hard time understanding the RetryExponential class that is used in conjunction with QueueClients (and I assume SubscriptionClients as well).

The properties are listed here, but I don't think my interpretation of their descriptions is correct.

Here's my interpretation...

    var minBackoff = TimeSpan.FromMinutes(5);  // wait 5 minutes for the first attempt?
    var maxBackoff = TimeSpan.FromMinutes(15);  // all attempts must be done within 15 mins?
    var deltaBackoff = TimeSpan.FromSeconds(30); // the time between each attempt?
    var terminationTimeBuffer = TimeSpan.FromSeconds(90); // the length of time each attempt is permitted to take?
    var retryPolicy = new RetryExponential(minBackoff, maxBackoff, deltaBackoff, terminationTimeBuffer, 10);

My worker role has only attempted to processing a message off the queue twice in the past hour even though I think based on the configuration above it should go off more frequently (every 30 seconds + whatever processing time was used during the previous attempted up to 90 seconds). I assume that these settings would force a retry every 2 mins. However, I don't see how this interpretation is Exponential at all.

Are my interpretations for each property (in comments above) correct? If not (and I assume they're not correct), what do each of the properties mean?

Wera

Answer 1

As you suspected, the values you included do not make sense for the meaning of these parameters. Here is my understanding of the parameters:

• DeltaBackoff - the interval to use to exponentially increment the retry interval by.
• MaximumBackoff - The maximum amount of times you want between retries.
• MaxRetryCount - the maximum amount of time the system will retry the operation.
• MinimalBackoff - the minimum amount of time you want between retries.
• TerminationTimeBuffer - the maximum amount of time the system will retry the operation before giving up.

It always will retry up to the maxRetryCount, in your case 10, unless the terminationTimeBuffer limit is hit first.

It will also not try for a period greater than the terminationTimeBuffer, which in your case is 90 seconds, regardless of it hasn't hit the max retry count yet.

The minBackoff is the minimal amount of time you will wait between retries and maxBackoff is the maximum amount of time you want to wait between retries.

The DeltaBackOff value is the value at which each retry internal will grow by exponentially . Note that this isn't an exact time. It randomly chooses a time that is a little less or a little more than this time so that multiple threads all retrying aren't doing so at the exact same time. Its randomness staggers this a little. Between the first actual attempt and the first retry there will be the minBackOff interval only. Since you set your deltaBackOff to 30 seconds, if it made it to a second retry it would be roughly 30 seconds plus the minBackOff. The third retry would be 90 seconds plus the minBackOff, and so on for each retry until it hits the maximum backoff.

One thing I would make sure to point out is that this is a retry policy, meaning if an operation receives an exception it will follow this policy to attempt it again. If operations such as Retrieve, Deadletter, Defer, etc. fail then this retry policy is what will kick in. These are operations against service bus, not exceptions in your own processing.

I could be wrong on this, but my understanding is that this isn't directly tied to the actual receipt of a message for processing unless the call to receive fails. Continuous processing is handled through the Receive method and your own code loop, or through using the OnMessage action (which behind the scenes also uses the Receive). As long as there isn't an error actually attempting to receive then this retry policy doesn't get applied. The interval used between calls to receive is set by either your own use of the Receive method which takes a timespan, or if you set the MessagingFactory.OperationTimeout before creating the queueClient object. If a receive call reaches it's limit of waiting either because you used the overload that provides a timespan on Receive or it hits the default, a Null is simply returned. This is not considered an exception and therefore the retry policy won't kick in.

Sadly, I think you have to code your own exponential back off for actual processing. There are tons of examples out there though.

And yes, you can set this retry policy on both QueueClient and SubscriptionClient.

Surface

Mike you assertion on Receive not participating in Retry is correct. A Receive operation will return null or a message and then will need to be called again for you to receive the next message. Retry policy is for handling errors and retrying those operations such as completing a messages, sending messages or creating a queue etc. – Quiff

@Surface every documents and people talk about MaximumBackoff but value? do you know about value? – Diantha

@NuriYILMAZ The MaximumBackoff is a timespan. The retry policy in place for Service bus by default is set by RetryPolicy.Default, which uses a 30 second MaximumBackoff. If you are setting this to something else, you want to use a value that has the longest amount of time you are willing to wait between checking the queue for work. – Surface

I think your understanding of deltaBackoff is incorrect. See the comments on the source: github.com/ProjectExtensions/ProjectExtensions.Azure.ServiceBus/… – Luciusluck

@Luciusluck Looking at the code at the end of the file you link to the deltaBackoff is used when deciding the amount of time to delay, and it uses that to get an actual amount of time between 80% or 120% of the deltaBackoff, then multiplies that by the number of tries it's on to determine the time to delay. I think that is what I've said above, but maybe not. Can you be more specific on where I'm misunderstanding something? Thanks! – Surface