The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › Splitting into services

Circuit breaker

If you keep calling a dead service, you go down too

The order service calls the payment service once per second. Partway through, the payment service stops responding for a while. When the failure rate goes over a set level, a circuit breaker blocks the calls themselves. This stops threads from being tied up waiting for a service that does not respond.

Checked against real execution · The call sequences of the 27 combinations were fed to Resilience4j 2.4.0 and checked against the results

Concepts in this lab

Circuit breaker

A mechanism that blocks calls to a service that has failed, so that callers do not keep calling it and waiting. When failures pass a certain level, it moves to the open state, which blocks calls, and makes later calls fail immediately. After a certain time, it lets some calls through as a trial to check whether the service has recovered. It stops a failure in one service from spreading to the services that call it.

Monolithic architecture and microservice architecture

A monolithic architecture builds all features as one application and deploys them together. A microservice architecture splits the features into small services, and each one is developed, deployed, and scaled independently. With microservices, each service can be deployed and scaled separately and failures are isolated, but the complexity of communication between services, data consistency, and operations grows a lot.

Bottleneck

The slowest part of a system, which limits the processing performance of the whole. No matter how much you improve the other parts, overall performance does not improve while this part stays the same. Performance improvement starts with finding it by measurement.

Try these

• Default settings — A 15s outage. When failures pile up, calls are blocked, and the circuit periodically tests whether the service has recovered.
• Without a circuit breaker — For the whole outage, every call waits 3s and then fails.
• Lower the threshold — 30%. It blocks calls sooner.
• A long open duration — 10s. Fewer trial calls are made during the outage.
• A short outage — A 5s outage with an 80% threshold. The failure rate does not reach the threshold, so the circuit never opens.

Running with the default settings

1. 1s: The call to the payment service succeeded.
2. 2s: The call to the payment service succeeded.
3. 3s: The call to the payment service succeeded.
4. 4s: The call to the payment service succeeded.
5. 5s: The call to the payment service succeeded.
6. 6s: The call to the payment service succeeded.
7. 7s: The call to the payment service succeeded.
8. 8s: The call to the payment service succeeded.
9. 9s: The call to the payment service succeeded.
10. 10s: The call to the payment service succeeded. Calls in the window: 10, failure rate 0%.
11. 11s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Calls in the window: 10, failure rate 10%.
12. 12s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Calls in the window: 10, failure rate 20%.
13. 13s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Calls in the window: 10, failure rate 30%.
14. 14s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Calls in the window: 10, failure rate 40%.
15. 15s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Among the last 10 calls, the failure rate became 50%, at or above the threshold of 50%. The circuit opens (OPEN). For the next 5s, calls to the payment service are blocked.
16. 16s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
17. 17s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
18. 18s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
19. 19s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
20. 20s: It has been 5s since the calls were blocked. To check whether the service has recovered, a few calls are let through as a trial (HALF_OPEN). The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Trial call 1/3.
21. 21s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. Trial call 2/3.
22. 22s: The payment service does not respond. The call waits 3s and fails with a timeout. During that time, one request thread was tied up. The failure rate of the 3 trial calls is 100%, at or above the threshold. The service is judged not recovered yet, so calls are blocked again (OPEN).
23. 23s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
24. 24s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
25. 25s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
26. 26s: The circuit is open (OPEN). The call is not sent to the payment service, and a fallback response is returned immediately. There is no waiting time.
27. 27s: It has been 5s since the calls were blocked. To check whether the service has recovered, a few calls are let through as a trial (HALF_OPEN). The call to the payment service succeeded. Trial call 1/3.
28. 28s: The call to the payment service succeeded. Trial call 2/3.
29. 29s: The call to the payment service succeeded. The failure rate of the 3 trial calls is 0%, below the threshold. The service is judged recovered, and the circuit returns to the normal state (CLOSED).
30. 30s: The call to the payment service succeeded.
31. 31s: The call to the payment service succeeded.
32. 32s: The call to the payment service succeeded.
33. 33s: The call to the payment service succeeded.
34. 34s: The call to the payment service succeeded.
35. 35s: The call to the payment service succeeded.
36. 36s: The call to the payment service succeeded.
37. 37s: The call to the payment service succeeded.
38. 38s: The call to the payment service succeeded.
39. 39s: The call to the payment service succeeded. Calls in the window: 10, failure rate 0%.
40. 40s: The call to the payment service succeeded. Calls in the window: 10, failure rate 0%.

Result of this run

• The state changed in the order CLOSED → OPEN → HALF_OPEN → OPEN → HALF_OPEN → CLOSED. While the circuit is open, calls are not made and a fallback response is returned right away, so no threads are tied up. The failing service is not put under extra load either, which gives it room to recover. After a set time, only a few calls are sent to check whether it has recovered.
• If the open duration is long, normal calls stay blocked for a while even after the service has recovered. If it is short, you test a service that has not recovered too often. Adjust the threshold and the open duration to the characteristics of the service.

Code

# application.yml (Resilience4j)
resilience4j.circuitbreaker.instances.payment:
  slidingWindowSize: 10                 # decide from the results of the last 10 calls
  minimumNumberOfCalls: 10
  failureRateThreshold: 50              # open the circuit when the failure rate is at or above this value
  waitDurationInOpenState: 5s          # how long the circuit stays open
  permittedNumberOfCallsInHalfOpenState: 3

@CircuitBreaker(name = "payment", fallbackMethod = "payLater")
PaymentResult pay(Order order) {
    return paymentClient.pay(order);     // times out after 3s if there is no response
}

PaymentResult payLater(Order order, Throwable t) {
    return PaymentResult.pending();      // runs instead when calls fail or are blocked
}

Assumptions of this model

This follows the behavior of Resilience4j with a count-based sliding window (the last 10 calls). All 27 combinations that use the circuit breaker were fed, in the same call order, to a real Resilience4j 2.4.0 circuit breaker (advancing the clock by 1 second at a time), and the numbers of successes, failures, and blocked calls and the order of state transitions were confirmed to match the screen. There is one difference. Resilience4j lets calls through as trial calls only after the wait duration has fully passed, so a call that arrives at exactly the same moment as the end of the wait duration is still blocked. The screen shows the result of treating the call as arriving a tiny bit after that moment. The circuit opens when the failure rate is at or above the threshold, and after the set time, the results of 3 trial calls decide whether to close it or open it again. It is simplified so that calls are made once per second and a service in an outage always fails with a 3s timeout. In real use, you can also use the slow call rate as a criterion, and there is a time-based sliding window too.

Previous lab: Kafka partitions and consumers · Next lab: Saga and outbox