Polly HandleTransientHttpError not catching HttpRequestException

Question

I've created a retry policy on my HttpClient in the Startup.ConfigureServices method. Note also that by default, asp.net core 2.1 logs 4 [Information] lines for each call made by the HttpClient which are shows in the logs at the end of my question.

services.AddHttpClient("ResilientClient")
            .AddPolicyHandler(
                Policy.WrapAsync(
                    PollyRetryPolicies.TransientErrorRetryPolicy(),
                    Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

The policy is defined as follows. Note that I write the retry attempt to logs, so I will know if the retry policy is invoked.

public static IAsyncPolicy < HttpResponseMessage > TransientErrorRetryPolicy() {
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or < TimeoutRejectedException > ()
        .WaitAndRetryAsync(sleepDurations: ExponentialBackoffPolicy.DecorrelatedJitter(3, SEED_DELAY, MAX_DELAY),
            onRetry: (message, timespan, attempt, context) => {
                context.GetLogger() ? .LogInformation($ "Retrying request to {message?.Result?.RequestMessage?.RequestUri} in {timespan.TotalSeconds} seconds. Retry attempt {attempt}.");
    });
}

HandleTransientHttpError() is a Polly extension that states in it's comments:

The conditions configured to be handled are: • Network failures (as System.Net.Http.HttpRequestException)

My httpclient usage is like this:

using (HttpResponseMessage response = await _httpClient.SendAsync(request)) 
{
    response.EnsureSuccessStatusCode();

    try 
    {
        string result = await response.Content.ReadAsStringAsync();
        if (result == null || result.Trim().Length == 0) {
            result = "[]";
        }
        return JArray.Parse(result);
    } catch (Exception ex) {
        _logger.LogInformation($ "Failed to read response from {url}. {ex.GetType()}:{ex.Message}");
        throw new ActivityException($ "Failed to read response from {url}.", ex);
    }
}

The following logs are captured:

[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: Start processing HTTP request GET https://api.au.... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Sending HTTP request GET https://api.au..... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Received HTTP response after 2421.8895ms - 200
[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: End processing HTTP request after 2422.1636ms - OK
    
Unknown error responding to request: HttpRequestException:
System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: The server returned an invalid or unrecognized response.

at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at nd_activity_service.Controllers.ActivityController.GetND(String url) in /codebuild/output/src251819872/src/src/nd-activity-service/Controllers/ActivityController.cs:line 561

The Http call succeeds, and I can see it returns 200 - OK. But then the HttpRequestException is thrown. I assume the policy is not being invoked because the HttpClient message pipeline has already resolved, as we can see it returned 200 - OK. So how is it throwing an exception outside of this?

And how do I handle it? Wrap another policy around the method that handles HttpRequestExceptions specifically?

This error does appear to be transient. It is a scheduled job and works the next time it is called.

Ransell

Tip: result == null || result.Trim().Length == 0 => string.IsNullOrWhitespace(result) – Baber

@Ransell Do you know which line throw the HRE? My educated guess is this await response.Content.ReadAsStringAsync – Spiderwort

Answer 1

Your policy is defined against the HttpClient not against the HttpResponseMessage.

So, the response.EnsureSuccessStatusCode() will not trigger retry even if you receive for example 428.

The HandleTransientHttpError will trigger retry if you receive 408 or 5XX status codes from the downstream system. And when the SendAsync throws the HttpRequestException

---

Because your exception StackTrace looks like this:

System.Net.Http.HttpRequestException: Error while copying content to a stream.

System.IO.IOException: The server returned an invalid or unrecognized response.

that's why my educated guess is that this exception is thrown by the HttpContent class while you try to read the response body (ReadAsStringAsync).

This will not trigger retry since you have defined your policy on the HttpClient.

---

If you want to retry in those cases as well when either the response.EnsureSuccessStatusCode() throws HRE or when the response.Content.ReadAsStringAsync() does then you have to wrap your whole http communication and response processing logic into a retry policy.

Let me show you how to do that.

First use a PolicyRegistry instead of AddPolicyHandler:

//services.AddHttpClient("ResilientClient")
//    .AddPolicyHandler(
//        Policy.WrapAsync(
//            TransientErrorRetryPolicy(),
//            Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

services.AddHttpClient("ResilientClient");
var registry = services.AddPolicyRegistry();
registry.Add("retry", Policy.WrapAsync(
            TransientErrorRetryPolicy(),
            Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

Then ask the DI for the register, for example:

private readonly IHttpClientFactory factory;
private readonly IReadOnlyPolicyRegistry<string> registry;

public TestController(IHttpClientFactory factory, IReadOnlyPolicyRegistry<string> registry)
{
    this.factory = factory;
    this.registry = registry;
}

Finally retrieve the combined policy and execute the http call:

var retryPolicy = registry.Get<IAsyncPolicy<HttpResponseMessage>>("retry");
await retryPolicy.ExecuteAsync(async () => await IssueRequest());

private async Task<HttpResponseMessage> IssueRequest()
{
    var _httpClient = factory.CreateClient("ResilientClient");
    HttpResponseMessage response = await _httpClient.GetAsync("http://httpstat.us/428");

    response.EnsureSuccessStatusCode();
    return response;
}

I've used the httpstat.us to simulate 428 response.

Spiderwort

I also suspected response.Content.ReadAsStringAsync() was causing the exception, however, it is wrapped in a try - catch block and would have logged "Failed to read response from {url}..." which did not appear in my logs. So it is failing earlier than that. It has to be either SendAsync or EnsureSuccessStatusCode causing the HttpRequestException. It cannot be EnsureSuccessStatusCode since the Http logger showed a 200 response. So the only thing left to cause it is the SendAsync method. This is why I expected the retry policy to capture the exception and retry. – Ransell

@Ransell don't you have a stacktrace? That would help to nail down the source of HRE. – Spiderwort

You're right the stack trace will be helpful. I have updated the logs in my question with the portion of the stack trace that should be helpful. It is indeed being thrown from within the HttpClient class. Since the policy I have registered on the HttpClient seems unable to handle this error, I will implement the wrapping policy you provided in your answer. Thanks very much. – Ransell