Check string content of response before retrying with Polly

Question

I'm working with a very flaky API. Sometimes I get 500 Server Error with Timeout, some other time I also get 500 Server Error because I gave it input that it can't handle

SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.

Both of these cases give me HttpRequestException but I can look into the reply message from the server and determine the cause of the exception. If it is a timeout error, I should try again. If it is a bad input I should re-throw the exception, because no amount of retries will fix the problem of bad data.

What I'd like to do with Polly is to check on response message before attempting to retry. But all the samples I've seen so far only included type of exception.

I've come up with this so far:

        HttpResponseMessage response = null;
        String stringContent = null;
        Policy.Handle<FlakyApiException>()
             .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
              async (exception, timeSpan, context) =>
            {
                response = await client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"));
                stringContent = await response.Content.ReadAsStringAsync();

                if (response.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("Timeout"))
                {
                    throw new FlakyApiException(stringContent);
                }
            });

Is there a better way to do this kind of checking?

Twoply

Answer 1

In general, you can configure Polly policies to respond to the results of an execution (not just an exception), for example check an HttpResponseMessage.StatusCode with a predicate. Examples here in the Polly readme.

There is not however an in-built way to configure a single Polly policy to respond additionally to the content of the response message. This is because (as your example shows) obtaining that content requires a second async call, which may itself raise network errors.

This tl;dr engenders complications about how to express (in a simple syntax) a single policy which manages two different async steps with potentially different error handling for each step. Prior related discussion on Polly github: comment welcome.

As such, where a sequence requires two separate async calls, the Polly team currently recommends expressing this as two separate policies, similar to the example in the end of this answer.

---

The particular example in your question may not work because the onRetryAsync delegate (throwing FlakyApiException) is not itself guarded by the policy. A policy only guards the execution of delegates executed through .Execute/ExecuteAsync(...).

---

One approach could be to use two policies, a retry policy which retries all typical http exceptions and status codes including 500s; then inside that a Polly FallbackPolicy which traps the status code 500 representing SqlDateTime overflow, and excludes that from being retried by rethrowing as some distinguishing exception (CustomSqlDateOverflowException).

        IAsyncPolicy<HttpResponseMessage> rejectSqlError = Policy<HttpResponseMessage>
            .HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
            .FallbackAsync(async (delegateOutcome, context, token) =>
            {
                String stringContent = await delegateOutcome.Result.Content.ReadAsStringAsync(); // Could wrap this line in an additional policy as desired.
                if (delegateOutcome.Result.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("SqlDateTime overflow"))
                {
                    throw new CustomSqlDateOverflowException(); // Replace 500 SqlDateTime overflow with something else.
                }
                else
                {
                    return delegateOutcome.Result; // render all other 500s as they were
                }
            }, async (delegateOutcome, context) => { /* log (if desired) that InternalServerError was checked for what kind */ });

        IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .OrResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
            .OrResult(r => /* condition for any other errors you want to handle */)
            .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
                async (exception, timeSpan, context) =>
                {
                    /* log (if desired) retry being invoked */
                });

        HttpResponseMessage response = await retryPolicy.WrapAsync(rejectSqlError)
            .ExecuteAsync(() => client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"), cancellationToken));

Jungian

Thank you for taking time to write this up. Sounds a lot more complex than I was hoping for. Let me wrap my head around it and give it a try – Twoply

@mountain-traveller Something to have in account: throwing an exception we're losing the original HTTP Status Code - I mean we're changing a final response from HTTP Status Code, to an Exception. E.g, imagine we're working with a 401, instead of a 500. Throwing an exception, and having a failure in all retries - assume 3 retries for example, at the end, you will get the custom exception you had defined, not the 401 in this example. – Untried

Is there any way to produce a retry without throwing the exception? After all retries, the response will be an exception, not the original HTTP status code that causes the retries. – Untried

Answer 2

If I understand your question correctly then you want to retry only if the status code is 500 and the body contains Timeout. If that's the case then you can define your policy just like this

Policy<HttpResponseMessage>
    .HandleResult(response =>
        response.StatusCode == System.Net.HttpStatusCode.InternalServerError
        && response.Content.ReadAsStringAsync().GetAwaiter().GetResult().Contains("Timeout"))
    .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt);

---

UPDATE #1

Just to clarify. Even tough .GetAwaiter().GetResult() should be avoided whenever possible, here I consider it as a valid use case to utilize it:

• There is no HandleResultAsync builder method, so we have to use HandleResult sync method here
• First we filter for 500 status code and then we lazily evaluate the response body
• I assumed the response body is fairly small due to the fact we should not expose too much information in case of Internal Server Error

Nevadanevai

But that's a blocking call. It won't scale. – Witha

@Witha If you can utilize async IO in your to-be-decorated method then use WaitAndRetryAsync instead of WaitAndRetry. – Nevadanevai

I meant the ReadAsStringAsync().GetAwaiter().GetResult() call. There is no HandleResultAsync() so HandleResult() has to be blocking. I've addressed it in my answer using HttpRequestMessage.Properties. – Witha

@Witha Ohh I got your point. In other cases I would agree with you but this use case is a bit special. First we filter for Internal Server error. If the status code matches then we search a predefined string in the body. Since Internal Server errors should not be really verbose to avoid leaking information I expect a short response body. Due to this a blocking call would not cause scalability issues IMHO. – Nevadanevai

Even with HttpCompletionOption.ResponseHeadersRead? See learn.microsoft.com/en-us/dotnet/api/… – Witha

@Witha Yet again, I assume that response body is not large (several hundreds of bytes or few kilobytes at most). Because of this assumption I don't consider using blocking call here is a bad practice. I would prefer to go with a simpler solution and refactor it to a more complex whenever the simpler causes issues. – Nevadanevai

Answer 3

For Http, I chose to solve this problem using DelegatingHandler (DH) pattern, and polly. There is no HandleResultAsync(), so the issue still exists for a generalized question.

With polly, I avoid a solution that has "coupling".

I've had great success with using a retry policy in a DelegatingHandler as it follows SRP, and provides a nice SoC (see this SO post). Here is the retry DH I use typically for reference.

For your question at hand, there are 2 things: retry, and conditions to retry on. Building on my retry DH, I exploded it into two DelegatingHandlers: a retry DH that retries on a "signal", and a latter retry signaling DH that signals a retry. HttpRequestMessage's .Properties (or .Options) bag is used to signal.

I find it easily maintainable, and is not complex by avoiding nested polly policies or blocking call. I have few APIs using the async request/reply pattern, so the retry DH (used for polling) is reusable (nugetized), and the retry signaling DH is different as per the API. You can obviously combine them into one by inlining the signaling code into the action arg.

HttpClient CoR (chain of responsibility):
... -> retry on signal DH -> retry signaling DH -> ...

Here is the retry signaling DH for your conditions to retry.

public class RetrySignalingOnConditionHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);

        // tweak conditions accordingly
        if (response.StatusCode == (HttpStatusCode)500)
        {
            request.Properties[RequestProperties.RetrySignal] = true;
            return response;
        }
        var content = await response.Content.ReadAsStringAsync(cancellationToken);
        if (content.Contains("Timeout"))
        {
            request.Properties[RequestProperties.RetrySignal] = true;
            return response;
        }

        return response;
    }
}

internal static class RequestProperties
{
    internal static string RetrySignal = nameof(RetrySignal);
}

Here is the retry DH that retries on the signal. It resets the signal before the attempt.

public class ExponentialBackoffRetryOnSignalHandler : DelegatingHandler
{
    private readonly IAsyncPolicy<(HttpRequestMessage request, HttpResponseMessage response)> retryPolicy;

    public ExponentialBackoffRetryOnSignalHandler(
        IRetrySettings retrySettings)
    {
        _ = retrySettings
            ?? throw new ArgumentNullException(nameof(retrySettings));

        var sleepDurations = Backoff.ExponentialBackoff(
            initialDelay: TimeSpan.FromMilliseconds(retrySettings.RetryDelayInMilliseconds),
            retryCount: retrySettings.RetryCount);

        retryPolicy = Policy
            .HandleResult<(HttpRequestMessage request, HttpResponseMessage response)>(tuple =>
                tuple.request.Properties.TryGetValue(RequestProperties.RetrySignal, out var retrySignaledObj) && (bool)retrySignaledObj)
            .WaitAndRetryAsync(
                sleepDurations: sleepDurations,
                onRetry: (responseResult, delay, retryAttempt, context) =>
                {
                    // note: response can be null in case of handled exception
                    responseResult.Result.response?.Dispose();
                });
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var tuple = await retryPolicy.ExecuteAsync(
            action: async (ct) =>
            {
                request.Properties.Remove(RequestProperties.RetrySignal);
                var response = await base.SendAsync(request, ct)
                    .ConfigureAwait(false);
                return (request, response);
            },
            cancellationToken: cancellationToken)
                .ConfigureAwait(false);
        return tuple.response;
    }
}

public interface IRetrySettings
{
    int RetryCount { get; }
    int RetryDelayInMilliseconds { get; }
}

Here is the full code that I use along with tests.

Witha

Answer 4

I would suggest making use of a single DelegatingHandler class that executes a single IAsyncPolicy within the DelegatingHandler's SendAsync method.

The response body can then be read asynchronously within an async lambda that's passed to the IAsyncPolicy's ExecuteAsync method.

public class MyRetryDelegatingHandler : DelegatingHandler
{
    #region Fields and constructor
    private readonly IAsyncPolicy<HttpResponseMessage> _retryPolicy;

    /// <param name="retryPolicy">The retryPolicy to execute during the SendAsync operation</param>
    public MyRetryDelegatingHandler(IAsyncPolicy<HttpResponseMessage> retryPolicy)
    {
        _retryPolicy = retryPolicy;
    }
    #endregion

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        // Execute retry policy
        var response = await _retryPolicy.ExecuteAsync(async () =>
        {
            // Send the HTTP request using base
            var response = await base.SendAsync(request, cancellationToken);

            if (response.StatusCode == HttpStatusCode.InternalServerError)
            {
                // Read the response body
                var content = await response.Content.ReadAsStringAsync();

                if (content is not null)
                {
                    // Check for timeout in response body
                    if (!content.Contains("Timeout"))
                    {
                        // Throw FlakyApiException if response was not a timeout error
                        throw new FlakyApiException(content);
                    }
                }
            }

            return response;
        });

        return response;
    }
}

The IAsyncPolicy can then be instantiated in Program.cs as follows. The IAsyncPolicy can also be instantiated within the DelegatingHandler.

// Create Polly policy
var teamsRetryPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()
    .WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(medianFirstRetryDelay: TimeSpan.FromMilliseconds(500), retryCount: 5));

// Add TeamsRetryDelegatingHandler to services
builder.Services.AddTransient(provider => new MyRetryDelegatingHandler(teamsRetryPolicy));

// Add a typed HttpClient
// The extension method registers the typed client and type as transient services
builder.Services.AddHttpClient<IMyService>()
    .AddHttpMessageHandler<MyRetryDelegatingHandler>();

The typed HttpClient can then be injected into the service class that was used as the type of the typed HttpClient. A very basic example can be seen below.

public class MyService
{
    #region Fields and constructor
    private readonly HttpClient _httpClient;

    public MyService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }
    #endregion

    public async Task<HttpResponseMessage> SendApiRequest()
    {
        return await _httpClient.GetAsync("<url>");
    }
}

public interface IMyService
{
    Task<HttpResponseMessage> SendApiRequest();
}

Romona