Polly WaitAndRetryAsync hangs after one retry

Asked 26/6, 2019 at 9:15 Answered 3/3, 2022 at 17:10

Solved c#.net polly retry-logic exponential-backoff

I'm using Polly in very basic scenario to do exponential backoff if an HTTP call fails:

protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
    return await HandleTransientHttpError()
        .Or<TimeoutException>()
        .WaitAndRetryAsync(4, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
        .ExecuteAsync(async () => await base.SendAsync(request, cancellationToken).ConfigureAwait(false));
}

private static PolicyBuilder<HttpResponseMessage> HandleTransientHttpError()
{
    return Policy
        .HandleResult<HttpResponseMessage>(response => (int)response.StatusCode >= 500 || response.StatusCode == System.Net.HttpStatusCode.RequestTimeout)
        .Or<HttpRequestException>();
}

I have a test API that just creates an HttpListener and loops in a while(true). Currently, I'm trying to test if the client retries correctly when receiving 500 for every single call.

while (true)
{
    listener.Start();
    Console.WriteLine("Listening...");
    HttpListenerContext context = listener.GetContext();
    HttpListenerRequest request = context.Request;

    HttpListenerResponse response = context.Response;
    response.StatusCode = (int)HttpStatusCode.InternalServerError;

    //Thread.Sleep(1000 * 1);
    string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
    byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
    response.ContentLength64 = buffer.Length;
    System.IO.Stream output = response.OutputStream;
    output.Write(buffer, 0, buffer.Length);
    output.Close();
    listener.Stop();
}

With the above code all works well and the retries happen after 3, 9, 27 and 81 seconds of waiting, respectively.

However, if I uncomment the Thread.Sleep call, the client retries once and then just hangs until the call times out for the other 3 retries, which is not the correct behavior.

The same thing also happens with the actual production API, which leads me to believe it's not a problem with my test API.

Umpire answered 26/6, 2019 at 9:15 Comment(6)

Try to ConfigureAwait(false) the awaiting of HandleTransientHttpError(). – Appolonia 26/6, 2019 at 9:54

@Theodor Zoulias: Unfortunately, that did not solve it. The context in which I'm executing this is similar to a console application, so I don't think it would make a difference theoretically either. – Umpire 26/6, 2019 at 10:11

Is this on .NET Framework? 4.6.1? It looks like an example of this issue: github.com/App-vNext/Polly/issues/642 . That issue is outside (not caused by) Polly - see discussion on .github.com/App-vNext/Polly/issues/642 and github.com/App-vNext/Polly/issues/658 – Goingson 26/6, 2019 at 11:52

@mountain traveller: It is on .NET Framework 4.6.1 indeed. – Umpire 26/6, 2019 at 12:6

@Tundor : Did you got any fix for it? – Pentahedron 22/11, 2019 at 11:28

@AstroBoy: Yes, I moved the Polly retry logic outside the actual http call, as Stephen Cleary suggests below. – Umpire 22/11, 2019 at 12:20

Using Polly within HttpClient doesn't work very well. A single SendAsync is intended to be a single call. I.e.:

Any HttpClient timeouts will apply to the single SendAsync call.
Some versions of HttpClient also dispose their content, so it can't be reused in the next SendAsync call.
As noted in the comments, this kind of hang is a known issue and cannot be fixed by Polly.

Bottom line: overriding SendAsync is great for adding pre-request and post-request logic. It's not the right place to retry.

Instead, use a regular HttpClient and have your Polly logic retry outside the GetStringAsync (or whatever) call.

Neutron answered 26/6, 2019 at 19:12 Comment(2)

Any thoughts on the workaround by disposing the results inside the RetryAsync callback? – Oldfashioned 3/3, 2022 at 17:16

@scuba88: I haven't tried it. As long as it doesn't close the underlying socket, it might work fine. – Neutron 3/3, 2022 at 18:23

This seems to be an appropriate workaround for the known issue with .NET Framework and using Polly within the HttpClient. We must dispose the result on retry in order to allow multiple requests. See discussion on the original issue here and another discussion describing the workaround here. I have only briefly tested this to determine that it works, but have not fully researched what side-effects might be present.

Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(msg => RetryableStatusCodesPredicate(msg.StatusCode))
.RetryAsync(retryCount, onRetry: (x, i) =>
{
    x.Result.Dispose(); // workaround for https://github.com/aspnet/Extensions/issues/1700
}));

Oldfashioned answered 3/3, 2022 at 17:10 Comment(0)

Recommended topics

Hot tags