Polly won't throw on some exceptions?

Question

I'm using Polly with .net Core. My ConfigureServices is :

private static void ConfigureServices()
{
    var collection = new ServiceCollection();
    var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(3);  
    collection.AddHttpClient<INetworkService, NetworkService>(s=>
             {
                   s.BaseAddress = new Uri("http://google.com:81"); //this is a deliberate timeout url
             })
    .AddPolicyHandler((a,b)=>GetRetryPolicy(b))
    .AddPolicyHandler(timeoutPolicy); ;
    ...
}

This is the GetRetryPolicy function:

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(HttpRequestMessage req)
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => httpStatusCodesWorthRetrying.Contains(msg.StatusCode))  // see below
        .Or<TimeoutRejectedException>()
        .Or<TaskCanceledException>()
        .Or<OperationCanceledException>()
        .WaitAndRetryAsync(3, retryAttempt =>
        {
            return TimeSpan.FromSeconds(3);
        }, onRetry: (response, delay, retryCount, context) =>
        {

            Console.WriteLine($"______PollyAttempt_____ retryCount:{retryCount}__FOR_BaseUrl_{req.RequestUri.ToString()}");
        });
}

Those are the httpcodes I want to retry :

static HttpStatusCode[] httpStatusCodesWorthRetrying = {
   HttpStatusCode.RequestTimeout, // 408
   HttpStatusCode.InternalServerError, // 500
   HttpStatusCode.BadGateway, // 502
   HttpStatusCode.ServiceUnavailable, // 503
   HttpStatusCode.GatewayTimeout // 504
};

Ok. And this is the actual invocation :

public async Task Work()
    {

        try
        {
            
            HttpResponseMessage response = await _httpClient.GetAsync("");
            Console.WriteLine("After work");

        }
        catch (TimeoutRejectedException ex)
        {
            Console.WriteLine("inside TimeoutRejectedException");
        }

        catch (Exception ex)
        {
            Console.WriteLine("inside catch main http");
        }
}

The output is :

_PollyAttempt retryCount:1__FOR_BaseUrl_http://google.com:81/
_PollyAttempt retryCount:2__FOR_BaseUrl_http://google.com:81/
_PollyAttempt retryCount:3__FOR_BaseUrl_http://google.com:81/
inside TimeoutRejectedException

(notice it throws) Which is OK. Because Polly throws after this invalid URL is timeout.

But if I change the http://google.com:81/ to an "internal server error" url : (this return 500)

https://run.mocky.io/v3/9f1b4c18-2cf0-4303-9136-bb67d54d0148

Then it doesn't throw but continues :

_PollyAttempt retryCount:1__FOR_BaseUrl_https://run.mocky.io/v3/9f1b4c18-2cf0-4303-9136-bb67d54d0148
_PollyAttempt retryCount:2__FOR_BaseUrl_https://run.mocky.io/v3/9f1b4c18-2cf0-4303-9136-bb67d54d0148
_PollyAttempt retryCount:3__FOR_BaseUrl_https://run.mocky.io/v3/9f1b4c18-2cf0-4303-9136-bb67d54d0148
After work

(notice "after work" at the end)

Question:

Why does Polly throw at timeout, But doesn't throw at another condition ? I explictly wrote : .OrResult(msg => httpStatusCodesWorthRetrying.Contains(msg.StatusCode)) and 500 is one of them.

Parkerparkhurst

Check if this helps. – Sarcoma

The whole OrResult part is not needed. The HandleTransientFailure covers 408 and 5xx status codes. – Ashlieashlin

I'm not sure about the TimeoutRejectedException was handled in the correct way as well. In the Work you are catching it and that means Retry policy won't be triggered because of TimeoutRejectedException. – Ashlieashlin

@peter after all attempts were tried without success ,polly throws timedoutexception. Imho – Parkerparkhurst

@RoyiNamir Yes, because you have an outer / global timeout policy. In other words the timeout will be your outer policy and the retry will be the inner. – Ashlieashlin

@peter so was i doing something wrong in trying to catch timeout? What will be the correct code? – Parkerparkhurst

@RoyiNamir It depends. You can define local / per request timeout and you can define one global timeout. In the latter case the timeout includes all requests (the initial attempt + retry attempts + penalties between retries) – Ashlieashlin

@RoyiNamir Here I have detailed the difference between the two kinds of timeout if you want to better understand. – Ashlieashlin

@Peter im sorry , i might be dumb here. But in my first example i do get timeout after all 3 attempts including first attempt has failed. So ill be happy if you can kindly post an answer to show me what im doing wrong. Goal : throw after all attempts were failed (timeout or 5xx,480) – Parkerparkhurst

@RoyiNamir I remembered that the handler registration order working in the other way around (firstly registered will be the inner...). Sorry, I was wrong. Firstly registered will be the utmost outer. I usually use PolicyWrap to avoid this kind of confusion. – Ashlieashlin

@RoyiNamir I've left a post to make it clear how does the two examples differ from each other. I hope it gives you clarity. – Ashlieashlin

Answer 1

This is expected behavior. A delegate invocation results in either an exception or a return value. When the Polly retries are done, then it propagates whatever result was last, whether it is an exception or a return value.

In this case, the response would have a 500 status code.

Openair

Answer 2

As @StephenCleary said that's how the Polly works.

First let me share with you the cleaned up version of your code
then I will give you some explanation about the observed behaviours.

ConfigureServices

var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(3,
    onTimeoutAsync: (_, __, ___) => {
        Console.WriteLine("Timeout has occured");
        return Task.CompletedTask;
});

services.AddHttpClient<INetworkService, NetworkService>(
    client => client.BaseAddress = new Uri("https://httpstat.us/500"))
.AddPolicyHandler((_, request) => Policy.WrapAsync(GetRetryPolicy(request), timeoutPolicy));

GetRetryPolicy

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(HttpRequestMessage req)
    => HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or<TimeoutRejectedException>()
        .Or<OperationCanceledException>()
        .WaitAndRetryAsync(3,
            _ => TimeSpan.FromSeconds(3),
            onRetry: (_, __, retryCount, ___) =>
                Console.WriteLine($"POLLY retryCount:{retryCount} baseUrl: {req.RequestUri}"));

---

Example http://google.com:81

1. Initial request has been sent out
2. No response has been received under 3 seconds
3. Timeout Policy triggered
4. TimeoutRejectedException is thrown
5. Retry policy is aware of that exception, so it triggers
6. Retry policy issues 3 seconds penalty
7. Retry policy issues a new request
8. No response has been received under 3 seconds
9. ...
   n. Retry policy is aware of that exception, but it has reached the max retry count
   n+1. Retry throws the exception that it could not handle, so in this case the TimeoutRejectedException

Example https://httpstat.us/500

1. Initial request has been sent out
2. A response with status code 500 has received under 3 seconds
3. Retry policy is aware of that status code, so it triggers
4. Retry policy issues 3 seconds penalty
5. Retry policy issues a new request
6. response with status code 500 has received under 3 seconds
7. ...
   n. Retry policy is aware of that status code, but it has reached the max retry count
   n+1. Retry returns with that response that it could not handle, so in this case the 500

Because there is a lack of EnsureSuccessStatusCode method call that's why no exception is being thrown.

As you can see in the second example the TimeoutPolicy is not triggered at all.

Ashlieashlin

BTW I must tell you that in my project we've built an AUTH docker api. and we made it fully REST. ( 200,409 (conflict) , 401.... etc etc....) and when we added Polly , I got hit by those non success code. I couldn't event do "EnsureSuccessCode" ( I konow it's not polly's but....). This is the last time I build pure REST api. next times I will return 200 and sub codes. – Parkerparkhurst

That's perfectly fine, I just mentioned EnsureSuccessStatusCode because it throws exception if status code is above 299. So, if you want to break the happy path then that's the easiest way. – Ashlieashlin

See also http-error-codes-are-retried-by-polly-net-by-default – Michelsen