Receiving error 'The request message was already sent' when using Polly

Question

I am currently using Polly to limit the number of requests I send. This is the policy I currently have:

private AsyncPolicyWrap<HttpResponseMessage> DefineAndRetrieveResiliencyStrategy()
{
    HttpStatusCode[] retryCodes = {
       HttpStatusCode.InternalServerError,
       HttpStatusCode.BadGateway,
       HttpStatusCode.GatewayTimeout
    };
    
    var waitAndRetryPolicy = Policy
        .HandleResult<HttpResponseMessage>(e => e.StatusCode == HttpStatusCode.ServiceUnavailable || e.StatusCode == (HttpStatusCode)429)
        .WaitAndRetryAsync(10,
            attempt => TimeSpan.FromSeconds(5), (exception, calculatedWaitDuration) =>
            {
                _log.Info($"Bitfinex API server is throttling our requests. Automatically delaying for {calculatedWaitDuration.TotalMilliseconds}ms");
            }
        );
    
    var circuitBreakerPolicyForRecoverable = Policy
        .Handle<HttpResponseException>()
        .OrResult<HttpResponseMessage>(r => retryCodes.Contains(r.StatusCode))
        .CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 3,
            durationOfBreak: TimeSpan.FromSeconds(3),
            onBreak: (outcome, breakDelay) =>
            {
                _log.Info($"Polly Circuit Breaker logging: Breaking the circuit for {breakDelay.TotalMilliseconds}ms due to: {outcome.Exception?.Message ?? outcome.Result.StatusCode.ToString()}");
                
            },
            onReset: () => _log.Info("Polly Circuit Breaker logging: Call ok... closed the circuit again"),
            onHalfOpen: () => _log.Info("Polly Circuit Breaker logging: Half-open: Next call is a trial")
        );
    
    return Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicyForRecoverable);
}

I have the following request sender:

private async Task<string> SendRequest(GenericRequest request, string httpMethod, string publicKey, string privateKey)
{
    var resiliencyStrategy = DefineAndRetrieveResiliencyStrategy();

    using (var client = new HttpClient())
    using (var httpRequest = new HttpRequestMessage(new HttpMethod(httpMethod), request.request))
    {
        string json = JsonConvert.SerializeObject(request);
        string json64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
        byte[] data = Encoding.UTF8.GetBytes(json64);

        client.BaseAddress = new Uri(Properties.Settings.Default.BitfinexUri);

        var hashMaker = new HMACSHA384(Encoding.UTF8.GetBytes(privateKey));
        byte[] hash = hashMaker.ComputeHash(data);
        string signature = GetHexString(hash);
        
        httpRequest.Headers.Add("X-BFX-APIKEY", publicKey);
        httpRequest.Headers.Add("X-BFX-PAYLOAD", json64);
        httpRequest.Headers.Add("X-BFX-SIGNATURE", signature);
        
        var message = await resiliencyStrategy.ExecuteAsync(() => client.SendAsync(httpRequest));
        var response = message.Content.ReadAsStringAsync().Result;

        return response;
    }
}

As soon as the code hits the waitAndRetryPolicy and awaits the required amount of time, I get the following error:

System.InvalidOperationException: 'The request message was already sent. Cannot send the same request message multiple times.'

I understand that this is happening because I am sending the same HttpRequest again but shouldn't the Polly library handle such an issue?

Whoop

Please prefer await message.Content.ReadAsStringAsync(); over message.Content.ReadAsStringAsync().Result. – Modesta

Answer 1

That exception:

System.InvalidOperationException: 'The request message was already sent. Cannot send the same request message multiple times.'

is thrown by the internals of HttpClient if you call directly into any .SendAsync(...) overload with an HttpRequestMessage which has already been sent.

If you are using .NET Core, the recommended solution is to use Polly with HttpClientFactory: this solves the above exception by executing the policy (for example retry) via a DelegatingHandler within HttpClient. It also solves the socket-exhaustion problem which can be caused by a frequent create/dispose of HttpClient, which the code posted in the question may be vulnerable to.

If you using .NET framework, the recommended solutions are:

• replicate the way HttpClientFactory places the policy in a DelegatingHandler; or
• refactor your code to manufacture a new instance of HttpRequestMessage (or clone the existing instance) within the code executed through the policy.

This stackoverflow question discusses the problem extensively and many variants on the above solutions.

Rightminded

Answer 2

with .NET Framework alternatively to make it generic you can clone request and store in contextData utilizing IAsyncPolicy.ExecuteAsync function second parameter IDictionary<string, object> contextData

Coupe

Answer 3

If you would split up your code into the following two functions:

private HttpRequestMessage CreateRequest(GenericRequest request, string httpMethod, string publicKey, string privateKey)
{
    var httpRequest = new HttpRequestMessage(new HttpMethod(httpMethod), request.request);
    httpRequest.Headers.Add("X-BFX-APIKEY", publicKey);

    string json = JsonConvert.SerializeObject(request);
    string json64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
    httpRequest.Headers.Add("X-BFX-PAYLOAD", json64);

    byte[] data = Encoding.UTF8.GetBytes(json64);
    var hashMaker = new HMACSHA384(Encoding.UTF8.GetBytes(privateKey));
    byte[] hash = hashMaker.ComputeHash(data);
    httpRequest.Headers.Add("X-BFX-SIGNATURE", GetHexString(hash));

    return httpRequest;
}

private async Task<string> SendRequest(GenericRequest request, string httpMethod, string publicKey, string privateKey)
{
    var message = await DefineAndRetrieveResiliencyStrategy().ExecuteAsync(async () =>
    {
        var httpRequest = CreateRequest(request, httpMethod, publicKey, privateKey);
        await client.SendAsync(httpRequest);
    });
    return await message.Content.ReadAsStringAsync();
}

then your problem would vanish.

• The CreateRequest is responsible to create a new HttpRequestMessage whenever it is called (either by the initial request or any further retry attempt)
• The SendRequest is responsible to decorate the downstream communication with the predefined strategy and parse the result of any
• The HttpClient setup should be done only once and reused many times

Modesta