Polly CircuitBreakerAsync is not working as I expect

Question

I'm just trying out the Polly CircuitBreakerAsync and it's not working as I expect.

What am I doing wrong here? I expect the code below to complete and say the circuit is still closed.

using Polly; 
using System;
using System.Threading.Tasks;

public class Program
{
    public static void Main(string[] args)
    {
        MainAsync(args).GetAwaiter().GetResult();
    }
    
    static async Task MainAsync(string[] args)
    {
        var circuitBreaker = Policy
            .Handle<Exception>()
            .CircuitBreakerAsync(
                3, // ConsecutiveExceptionsAllowedBeforeBreaking,
                TimeSpan.FromSeconds(5) // DurationOfBreak
            );

        Console.WriteLine("Circuit state before execution: " + circuitBreaker.CircuitState);

        await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
        await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
        await circuitBreaker.ExecuteAsync(() => { throw new System.Exception(); });
        await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
        await circuitBreaker.ExecuteAsync(() => Task.Delay(25));

        Console.WriteLine("Circuit state after execution: " + circuitBreaker.CircuitState);
    }
}

Fiddle: https://dotnetfiddle.net/unfKsC

Output:

Circuit state before execution: Closed
Run-time exception (line 25): Exception of type 'System.Exception' was thrown.

Stack Trace:

[System.Exception: Exception of type 'System.Exception' was thrown.]
   at Program.<MainAsync>b__2() :line 25
   at Polly.Policy.<>c__DisplayClass116_0.<ExecuteAsync>b__0(Context ctx, CancellationToken ct)
   at Polly.CircuitBreakerSyntaxAsync.<>c__DisplayClass4_1.<<CircuitBreakerAsync>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Polly.CircuitBreaker.CircuitBreakerEngine.<ImplementationAsync>d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Polly.Policy.<ExecuteAsync>d__135.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Program.<MainAsync>d__a.MoveNext() :line 25
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Program.Main(String[] args) :line 9

Opinion

FYI static async Task Main(string[] args) was introduced in C# 7.1. – Angeliaangelic

Answer 1

This is working as expected

https://github.com/App-vNext/Polly/wiki/Circuit-Breaker

Exception handling

A circuit-breaker exists as a measuring-and-breaking device: to measure handled exceptions thrown by actions you place through it, and to break when the configured failure threshold is exceeded.

• A circuit-breaker does not orchestrate retries.
• A circuit-breaker does not (unlike retry) absorb exceptions. All exceptions thrown by actions executed through the policy (both exceptions handled by the policy and not) are intentionally rethrown. Exceptions handled by the policy update metrics governing circuit state; exceptions not handled by the policy do not.

In short it doesn't handle your exceptions, it rethrows them

Shalon

Thank you, I completely overlooked this. RTFM :) – Opinion

Answer 2

Circuit Breaker in general

Your code works as expected. The circuit breaker itself will not break because you have set the consecutive error count to 3. It means that if you have 3 successive failed calls then it will transition from Closed state to Open. If you try to execute yet another call then it will throw a BrokenCircuitException. In Closed state if an exception has been thrown and the threshold has not been reached then it re-throws the exception.

I always suggest to consider the Circuit Breaker as a proxy. It allows calls if everything works fine. If the consumed sub-system / sub-component seems like malfunctioning then it will prevent further calls to avoid unnecessary load.

Callback functions for debugging

When you define a Circuit Breaker policy then you can specify 3 callbacks:

• onBreak: When it transitions from Closed or HalfOpen to Open
• onReset: When it transitions from HalfOpen to Close
• onHalfOpen: When it transitions from Open to HalfOpen

The amended policy declaration:

var circuitBreaker = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(3, TimeSpan.FromSeconds(5), 
        onBreak: (ex, @break) => Console.WriteLine($"{"Break",-10}{@break,-10:ss\\.fff}: {ex.GetType().Name}"),
        onReset: () => Console.WriteLine($"{"Reset",-10}"),
        onHalfOpen: () => Console.WriteLine($"{"HalfOpen",-10}")
    );

Successive failure count

Let's change the consecutive fail threshold to 1 and let's wrap your ExecuteAsync calls in a try catch:

var circuitBreaker = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(1, TimeSpan.FromSeconds(5), 
        onBreak: (ex, @break) => Console.WriteLine($"{"Break",-10}{@break,-10:ss\\.fff}: {ex.GetType().Name}"),
        onReset: () => Console.WriteLine($"{"Reset",-10}"),
        onHalfOpen: () => Console.WriteLine($"{"HalfOpen",-10}")
    );

Console.WriteLine("Circuit state before execution: " + circuitBreaker.CircuitState);

try
{
    await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
    await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
    await circuitBreaker.ExecuteAsync(() => { throw new System.Exception(); });
    await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
    await circuitBreaker.ExecuteAsync(() => Task.Delay(25));
}
catch (Exception ex)
{
    Console.WriteLine("Circuit state after execution: " + circuitBreaker.CircuitState);
    Console.WriteLine(ex.GetType().Name);
}

Console.WriteLine("Circuit state after execution: " + circuitBreaker.CircuitState);

The output will be the following:

Circuit state before execution: Closed
Break     05.000    : Exception
Circuit state after execution: Open
Exception

As you can see the Circuit Breaker has broken and go from Closed to Open state. It has rethrown your exception.

Combine Retry and Circuit Breaker

In order to easily demonstrate when the CB throws BrokenCircuitException I will use a Retry logic around the CB.

var retry = Policy
    .Handle<Exception>()
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync(
        retryCount: 1,
        sleepDurationProvider: _ => TimeSpan.FromSeconds(1),
        onRetry: (exception, delay, context) =>
        {
            Console.WriteLine($"{"Retry",-10}{delay,-10:ss\\.fff}: {exception.GetType().Name}");
        });

This policy will try to re-execute your delegate either when an Exception or when a BrokenCircuitException has been thrown. It does that with 1 second delay between the initial attempt and the first (and only) retry.

Let's combine the two policies and let's amend the ExecuteAsync call:

var strategy = Policy.WrapAsync(retry, circuitBreaker);
try
{
    await strategy.ExecuteAsync(() => { throw new System.Exception(); });
}
catch (Exception ex)
{
    Console.WriteLine("Circuit state after execution: " + circuitBreaker.CircuitState);
    Console.WriteLine(ex.GetType().Name);
}

The output will be the following:

Circuit state before execution: Closed
Break     05.000    : Exception
Retry     01.000    : Exception
Circuit state after execution: Open
BrokenCircuitException

1. The initial call fails and it throws an Exception
2. CB breaks because the threshold has been reached and it re-throws the exception
3. The combined policy will escalate the problem from the CB to the Retry
4. Retry handles Exception that's why it waits a second before it tries to re-execute the delegate again
5. Retry tries to call the delegate again but it fails because the CB is Open that's why a BrokenCircuitException is thrown
6. Because there is no further retry that's why the retry policy will re-throw its exception (which is now a BrokenCircuitException instance)
7. That exception is catched by our catch block.

---

Fine-tuned example

Let's amend the parameters of these policies a bit:

• CB's durationOfBreak from 5 seconds to 1.5
• Retry's retryCount from 1 to 2

var retry = Policy
    .Handle<Exception>()
    .Or<BrokenCircuitException>()
    .WaitAndRetryAsync(2, _ => TimeSpan.FromSeconds(1),
        onRetry: (exception, delay, context) =>
        {
            Console.WriteLine($"{"Retry",-10}{delay,-10:ss\\.fff}: {exception.GetType().Name}");
        });

var circuitBreaker = Policy
    .Handle<Exception>()
    .CircuitBreakerAsync(1, TimeSpan.FromMilliseconds(1500),
        onBreak: (ex, @break) => Console.WriteLine($"{"Break",-10}{@break,-10:ss\\.fff}: {ex.GetType().Name}"),
        onReset: () => Console.WriteLine($"{"Reset",-10}"),
        onHalfOpen: () => Console.WriteLine($"{"HalfOpen",-10}")
    );

Console.WriteLine("Circuit state before execution: " + circuitBreaker.CircuitState);

var strategy = Policy.WrapAsync(retry, circuitBreaker);
try
{
    await strategy.ExecuteAsync(() => { throw new System.Exception(); });
}
catch (Exception ex)
{
    Console.WriteLine("Circuit state after execution: " + circuitBreaker.CircuitState);
    Console.WriteLine(ex.GetType().Name);
}

The output will be the following:

Circuit state before execution: Closed
Break     01.500    : Exception
Retry     01.000    : Exception
Retry     01.000    : BrokenCircuitException
HalfOpen
Break     01.500    : Exception
Circuit state after execution: Open
Exception

I hope this little demo app helped you to better understand how does Circuit Breaker work.

Ringer

Thank you for this comprehensive answer, it's very useful. – Opinion

I didn't understand the output of the last example... the first 3 lines i do understand. I don't understand why the 4th line is not the half open.. we first got on exception on the break - 2nd line, then waited a second for a retry - 3rd line. Now I would expect that after another 0.5 a second we would have reached the durationOfBreak and the cb will now transition to half open BEFORE the next retry which still has another 0.5 a second after that. Why wasn't it like that? – Sagittate

@YonatanNir The onRetry callback is called BEFORE the sleep. So it tells you that I'll try to perform a new retry attempt 1 second later. That's why the 3rd line is printed before HalfOpen – Ringer

@YonatanNir Please take a look at this sample application here I've used better reporting to print out the elapsed time as well. I hope it helps you to have clarity about the order of the events. – Ringer

@PeterCsala thanks I got it. Still got the original question where you answered me – Sagittate

Thank you for helping me understand it all! I was struggling to catch the BrokenCircuitException, and now I do understand what I did wrong. This is an example of a great answer here on SO! – Supervise