Hystrix Configuration

Question

I am trying to implement hystrix for my application using hystrix-javanica.

I have configured hystrix-configuration.properties as below

hystrix.command.default.execution.isolation.strategy=SEMAPHORE
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=10000 
hystrix.command.default.fallback.enabled=true
hystrix.command.default.circuitBreaker.enabled=true
hystrix.command.default.circuitBreaker.requestVolumeThreshold=3 
hystrix.command.default.circuitBreaker.sleepWindowInMilliseconds=50000
hystrix.command.default.circuitBreaker.errorThresholdPercentage=50

short-circuit pattern is working fine but i have a doubt in this hystrix.command.default.circuitBreaker.requestVolumeThreshold=3

1. Is it stating open the circuit after 3 failures or
2. Open the circuit after 3 concurrent failures.

Gone through the documentation link

Can anybody answer?

 

 

Perineurium

Answer 1

How Hystrix Circuit-Breaker operates: Hystrix does not offer a circuit breaker which breaks after a given number of failures. The Hystrix circuit will break if:

within a timespan of duration metrics.rollingStats.timeInMilliseconds, the percentage of actions resulting in a handled exception exceeds errorThresholdPercentage, provided also that the number of actions through the circuit in the timespan is at least requestVolumeThreshold

---

What is requestVolumeThreshold? requestVolumeThreshold is a minimum threshold for the volume (number) of calls through the circuit that must be met (within the rolling window), before the circuit calculates a percentage failure rate at all. Only when this minimum volume (in each time window) has been met, will the circuit compare the failure proportion of your calls against the errorThresholdPercentage you have configured.

Imagine there was no such minimum-volume-through-the-circuit threshold. Imagine the first call in a time window errors. You would have 1 of 1 calls being an error, = 100% failure rate, which is higher than the 50% threshold you have set. So the circuit would break immediately.

The requestVolumeThreshold exists so that this does not happen. It's effectively saying, the error rate through your circuit isn't statistically significant (and won't be compared against errorThresholdPercentage) until at least requestVolumeThreshold calls have been received in each time window.

Nijinsky

Bit confused with the rolling window. Can you please explain bit more? According to my configuration circuit should be open after 3 failures? Isn't it? – Perineurium

See github.com/Netflix/Hystrix/wiki/How-it-Works#circuit-breaker for more detail on how the circuit breaker works. The 3 you have configured is not a number of failures on which to break. Hystrix breakers break on percentage of errors (the errorThresholdPercentage=50% you have configured), considered across calls in a given time window. The requestVolumeThreshold=3 is (per my original answer) a minimum-volume-threshold of calls, which must be met in the same time window, to make the %age calculations statistically significant. – Nijinsky

See github.com/Netflix/Hystrix/wiki/… for how the time windows is configured. – Nijinsky

For a circuit-breaker operating by the same principles, concisely explained, you can also see here: github.com/App-vNext/Polly/wiki/… Though the configuration terminology used is different (and it's a .NET not Java product), the operating principles are the same. The Polly page also offers some explanations of how the various configuration parameters interact (eg what happens if you set them too high, too low etc) – Nijinsky

@mountain traveller: My first request error itself is calling my fallback method. I dint configure "requestVolumeThreshold" explicitly. Can you help me? I have just followed spring.io/guides/gs/circuit-breaker – Rawdan

Answer 2

I am rather new to hystrix but I guess I can help you.

In general hystrix.command.default.circuitBreaker.requestVolumeThreshold is a property that sets the minimum number of requests in a rolling window that will trip the circuit and its default value is 20 and its value can be changed in properties file or in our @HystrixCommand annotated method.

For example, if that property value is 20, then if only 19 requests are received in the rolling window (say a window of 10 seconds) the circuit will not trip open even if all 19 failed. If the failed request value reaches 20, then the circuit will be opened and the corresponding calls will be sent to fallback even if the call succeeds, till the sleeping window time period complete.

Sleeping window time period sets the amount of time, after tripping the circuit, to reject requests before allowing attempts again to determine if the circuit should again be closed. Its value is defaulted to 5000 milliseconds. This can be changed by overriding circuitBreaker.sleepWindowInMilliseconds property.

You can find all the properties and its description here.

Sheathing

You may want to look at: github.com/Netflix/Hystrix/wiki/…. Your answer is incorrect, in taking no account of errorThresholdPercentage. Hope this helps. See github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/… and github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/… – Nijinsky

Is there any way to verify those configuration properties to make sure these properties are really applied to configure HystrixCommand? – Flaherty