Circuit Breaker

Metadata

1. Overview

What problem does it solve?

In a microservices architecture, services call other services. Service A calls Service B, which calls Service C. If Service C goes down, Service B might hang for 30 seconds waiting for a timeout. While Service B is hanging, it consumes CPU and memory threads. Soon, Service B runs out of resources and crashes. Now Service A crashes.

A single failing service has caused a Cascading Failure that takes down the entire system.

The Circuit Breaker pattern solves this. It wraps the network call to Service C. If it detects that Service C is failing too often, it "trips" the circuit. Instead of making the network call and waiting 30 seconds, it instantly returns an error (or a fallback value) to Service B.

What breaks without it?

• Cascading Failures: One small component dying takes down the entire application.
• Resource Exhaustion: Threads and connection pools fill up waiting on dead upstream services.
• Thundering Herds: When a struggling service finally comes back online, it is instantly crushed by the backlog of retries, killing it again.

2. Motivation

The pattern borrows its name from electrical engineering. An electrical circuit breaker detects if too much current is flowing through a wire (which could cause a fire) and "trips" (breaks the circuit) to stop the electricity. You then have to manually reset it.

In software, Michael Nygard popularized the pattern in his legendary book Release It! (2007). Netflix later made it ubiquitous by open-sourcing their implementation, Hystrix, which they used to survive AWS outages.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

The State Machine

A Circuit Breaker operates as a state machine with three states:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffcc00', 'edgeLabelBackground':'#ffffff'}}}%%
stateDiagram-v2
    direction LR
    
    state "CLOSED (Normal)" as Closed
    state "OPEN (Failing)" as Open
    state "HALF-OPEN (Testing)" as HalfOpen

    Closed --> Open : Failure Rate > Threshold
    Open --> HalfOpen : Timeout Expires
    HalfOpen --> Closed : Success
    HalfOpen --> Open : Failure

1. CLOSED (Normal Operation): Current flows. The circuit breaker allows requests to pass through to the downstream service. It counts the number of failures (e.g., 500 errors, timeouts).
2. OPEN (Circuit Tripped): The wire is cut. If the failure rate exceeds a threshold (e.g., 5 errors in 10 seconds), the circuit opens. All requests instantly fail without attempting to hit the downstream service.
3. HALF-OPEN (Recovery Test): After a specific wait time (e.g., 30 seconds), the circuit lets one request pass through.
   • If it succeeds, the service is healthy again! Transition to CLOSED.
   • If it fails, the service is still down. Transition back to OPEN and restart the timer.

6. Internal Working

Fallbacks

When a circuit is OPEN, it doesn't just have to throw an error. It can return a Fallback.

• Static Fallback: Return a hardcoded default (e.g., UI shows a default avatar if the image service is down).
• Cache Fallback: Return the last known good data from Redis, even if it's slightly stale.
• Graceful Degradation: Disable a specific feature but keep the app running. (e.g., Amazon hides the "People who bought this also bought..." widget if the ML service is down, but you can still buy the item).

Sliding Windows

To track the failure rate, circuit breakers use a sliding window (just like a Rate Limiter).

• Count-based: Look at the last 100 requests. If 50 failed, trip.
• Time-based: Look at the last 10 seconds. If 50% failed, trip.

7. Implementation

Why Python? Python's decorators make it incredibly elegant to wrap any fragile network call with a Circuit Breaker without altering the core business logic of the function.

"""
028 - Circuit Breaker
A State Machine implementation of the Circuit Breaker pattern
using Python decorators.
"""
import time
from functools import wraps

# Circuit Breaker States
CLOSED = "CLOSED"
OPEN = "OPEN"
HALF_OPEN = "HALF_OPEN"

class CircuitBreakerOpenException(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_timeout=5):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        
        self.state = CLOSED
        self.failures = 0
        self.last_failure_time = None

    def __call__(self, func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            # 1. Check state before execution
            if self.state == OPEN:
                # Is it time to test recovery?
                if time.time() - self.last_failure_time > self.recovery_timeout:
                    print("[CircuitBreaker] Timeout expired. Entering HALF-OPEN state to test.")
                    self.state = HALF_OPEN
                else:
                    print("[CircuitBreaker] Circuit is OPEN. Fast-failing request.")
                    raise CircuitBreakerOpenException("Circuit is OPEN. Service unavailable.")

            # 2. Execute the function
            try:
                result = func(*args, **kwargs)
            except Exception as e:
                # 3. Handle Failure
                self.record_failure()
                raise e

            # 4. Handle Success
            self.record_success()
            return result

        return wrapper

    def record_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        print(f"[CircuitBreaker] Failure recorded ({self.failures}/{self.failure_threshold})")
        
        if self.state == HALF_OPEN:
            print("[CircuitBreaker] Test failed. Returning to OPEN state.")
            self.state = OPEN
            return
            
        if self.failures >= self.failure_threshold:
            print("[CircuitBreaker] Threshold reached! TRIPPING CIRCUIT to OPEN state.")
            self.state = OPEN

    def record_success(self):
        if self.state == HALF_OPEN:
            print("[CircuitBreaker] Test succeeded! Resetting to CLOSED state.")
            self.state = CLOSED
            self.failures = 0
        elif self.state == CLOSED and self.failures > 0:
            # Optional: Gradually heal failures in CLOSED state
            self.failures = 0


# ── Test Harness ──

# Wrap our fragile network call with the Circuit Breaker
breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=3)

# Simulate a fragile downstream service
is_service_down = True

@breaker
def fetch_user_data():
    if is_service_down:
        raise ConnectionError("Timeout connecting to database")
    return {"id": 1, "name": "Alice"}


if __name__ == "__main__":
    print("--- 1. Service is failing. Sending requests. ---")
    for i in range(4):
        try:
            print(f"\nRequest {i+1}:")
            fetch_user_data()
        except Exception as e:
            print(f"Error caught: {type(e).__name__}")

    print("\n--- 2. Waiting for recovery timeout (3s) ---")
    time.sleep(3.1)
    
    print("\n--- 3. Testing HALF-OPEN state (Service still down) ---")
    try:
        fetch_user_data()
    except Exception as e:
        print(f"Error caught: {type(e).__name__}")

    print("\n--- 4. Waiting again (3s) and fixing the service ---")
    time.sleep(3.1)
    is_service_down = False # The database is fixed!
    
    print("\n--- 5. Testing HALF-OPEN state (Service is UP!) ---")
    print("Result:", fetch_user_data())
    
    print("\n--- 6. Circuit is CLOSED again. Next request works normally. ---")
    print("Result:", fetch_user_data())

Sample Output

--- 1. Service is failing. Sending requests. ---

Request 1:
[CircuitBreaker] Failure recorded (1/3)
Error caught: ConnectionError

Request 2:
[CircuitBreaker] Failure recorded (2/3)
Error caught: ConnectionError

Request 3:
[CircuitBreaker] Failure recorded (3/3)
[CircuitBreaker] Threshold reached! TRIPPING CIRCUIT to OPEN state.
Error caught: ConnectionError

Request 4:
[CircuitBreaker] Circuit is OPEN. Fast-failing request.
Error caught: CircuitBreakerOpenException

--- 2. Waiting for recovery timeout (3s) ---

--- 3. Testing HALF-OPEN state (Service still down) ---
[CircuitBreaker] Timeout expired. Entering HALF-OPEN state to test.
[CircuitBreaker] Failure recorded (4/3)
[CircuitBreaker] Test failed. Returning to OPEN state.
Error caught: ConnectionError

--- 4. Waiting again (3s) and fixing the service ---

--- 5. Testing HALF-OPEN state (Service is UP!) ---
[CircuitBreaker] Timeout expired. Entering HALF-OPEN state to test.
[CircuitBreaker] Test succeeded! Resetting to CLOSED state.
Result: {'id': 1, 'name': 'Alice'}

--- 6. Circuit is CLOSED again. Next request works normally. ---
Result: {'id': 1, 'name': 'Alice'}

8. Complexity

9. Trade-offs

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. What is a Cascading Failure and how does a Circuit Breaker prevent it? Hint: When one service dies, callers hang, run out of threads, and die too. The breaker fast-fails to prevent callers from hanging.

2. Explain the three states of a Circuit Breaker. Hint: Closed (flowing), Open (fast-fail), Half-Open (send a single ping to test if it's fixed).

3. What is the difference between a Retry and a Circuit Breaker? Hint: They are opposites. Retry says "It failed, push harder!" Circuit breaker says "It failed, stop pushing to give it time to breathe!" (They are usually used together).

4. Give an example of "Graceful Degradation" as a fallback. Hint: If the ML recommendation engine is down, fall back to returning a static list of the Top 10 Most Popular movies instead of throwing an error.

13. Used By (Downstream Blocks)

• 029 Retry & Exponential Backoff — Retries without a circuit breaker will DDoS a struggling system. They must be paired.
• 035 Service Mesh — Service meshes implement circuit breakers proxy-side so developers don't have to write them in code.

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: Smart Exceptions

Modify the Python CircuitBreaker so it accepts an ignore_exceptions list. Create a UserNotFoundError and ensure that if the function raises it, the circuit breaker completely ignores it and doesn't increment the failure count.

Exercise 2: Fallback Function

Modify the decorator to accept a fallback_func. If the circuit is OPEN, instead of raising CircuitBreakerOpenException, it should execute and return the result of the fallback_func.

Website Metadata