Distributed Transactions (Sagas)

Metadata

1. Overview

What problem does it solve?

In a monolithic SQL database, if you need to deduct 100 from Alice and add 100 to Bob, you wrap it in an ACID Transaction (BEGIN; ... COMMIT;). If either step fails, the database automatically rolls back, ensuring money isn't created or destroyed.

But in a Microservices architecture, data is spread across multiple independent databases. If a user books a vacation, you must:

1. Book the Flight (Flight DB)
2. Book the Hotel (Hotel DB)
3. Charge the Card (Payment DB)

You cannot use a standard SQL COMMIT across three different databases over an HTTP network. If the flight books successfully, but the hotel is sold out, how do you instantly "undo" the flight booking?

Distributed Transactions are patterns (like Two-Phase Commit or Sagas) used to maintain data consistency across multiple independent services.

What breaks without it?

• Partial Execution: The user's credit card is charged $2000, but the flight fails to book. The system is permanently inconsistent, and the user is furious.
• Lost Updates: Two services try to update the same distributed entity simultaneously without locking.

2. Motivation

In the 1990s, the X/Open consortium created the XA standard for Two-Phase Commit (2PC). It worked well for tightly coupled, on-premise Enterprise databases (like Oracle communicating with IBM DB2).

However, 2PC is a blocking, synchronous protocol. If a network goes down during 2PC, databases can be locked indefinitely. As companies moved to highly scalable, asynchronous microservices, 2PC became a major bottleneck. The industry shifted to the Saga Pattern (originally described in a 1987 paper by Hector Garcia-Molina) which breaks long-running transactions into a sequence of smaller, independent local transactions.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

The Saga Pattern (Orchestration)

A Saga is a sequence of local database transactions. If one step fails, the Saga executes Compensating Transactions (undos) for all the steps that already succeeded.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffcc00', 'edgeLabelBackground':'#ffffff'}}}%%
graph TD
    classDef orchestrator fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef svc fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef undo fill:#f8d7da,stroke:#dc3545,stroke-width:2px;

    O[Saga Orchestrator]:::orchestrator
    
    O -->|1. Book Flight| F[Flight Service]:::svc
    O -->|2. Book Hotel| H[Hotel Service]:::svc
    O -->|3. Charge Card| P[Payment Service]:::svc
    
    P -.->|FAIL: Card Denied| O
    
    O -.->|4. Cancel Hotel| H_undo[Hotel Service<br/>(Compensating)]:::undo
    O -.->|5. Cancel Flight| F_undo[Flight Service<br/>(Compensating)]:::undo

Two-Phase Commit (2PC)

A strict, synchronous protocol managed by a Transaction Coordinator.

1. Phase 1 (Prepare): Coordinator asks all databases: "Are you ready to commit?" Databases lock their rows and reply "Yes."
2. Phase 2 (Commit): If all say "Yes", Coordinator says "Commit!". If any say "No", Coordinator says "Abort!".

The massive flaw: If the Coordinator crashes between Phase 1 and 2, the databases are stuck holding their row locks forever.

6. Internal Working

Saga: Choreography vs Orchestration

Like Event-Driven Architecture, Sagas can be implemented two ways:

1. Choreographed Saga:

   • Services publish events to Kafka.
   • FlightBooked triggers the Hotel service.
   • HotelFailed triggers the Flight service to run its own rollback.
   • Pros: No single point of failure.
   • Cons: "Spaghetti Architecture." Hard to understand the full flow. If a cycle occurs, it loops forever.

2. Orchestrated Saga:

   • A central service (The Orchestrator) tells the others what to do via APIs or message queues.
   • The Orchestrator maintains a State Machine (e.g., using AWS Step Functions or Netflix Conductor).
   • Pros: Very easy to see the status of a transaction. Easy to handle complex rollback logic.
   • Cons: The orchestrator becomes a bottleneck.

Compensating Transactions

You cannot simply run ROLLBACK. The Flight Service actually committed the booking to its database. A Compensating Transaction is a brand new transaction that reverses the effect.

• Original: INSERT INTO flights (user_id) VALUES (1)
• Compensation: UPDATE flights SET status = 'cancelled' WHERE user_id = 1

Crucial Requirement: Because compensating transactions might fail due to network errors, they must be wrapped in Retry & Backoff and they must be Idempotent.

7. Implementation

Why Python? We can easily model the Saga Orchestrator as a State Machine in Python, demonstrating how it traps errors and walks backwards through the compensation steps.

"""
026 - Distributed Transactions (Saga Orchestration)
A Python implementation of an Orchestrated Saga handling a Trip Booking.
If the Payment step fails, it automatically rolls back the Hotel and Flight.
"""
import time

# ── Mock Microservices ──

class FlightService:
    def book(self):
        print("  [Flight] ✈️  Booking flight...")
        time.sleep(0.5)
        return True
        
    def cancel(self):
        print("  [Flight] ❌ Cancelling flight (Compensation)...")

class HotelService:
    def book(self):
        print("  [Hotel] 🏨 Booking hotel room...")
        time.sleep(0.5)
        return True
        
    def cancel(self):
        print("  [Hotel] ❌ Cancelling hotel room (Compensation)...")

class PaymentService:
    def __init__(self, should_fail=False):
        self.should_fail = should_fail

    def charge(self):
        print("  [Payment] 💳 Charging credit card...")
        time.sleep(0.5)
        if self.should_fail:
            raise Exception("Insufficient Funds!")
        return True


# ── The Saga Orchestrator ──

class TripBookingSaga:
    def __init__(self, payment_fails=False):
        self.flight_svc = FlightService()
        self.hotel_svc = HotelService()
        self.payment_svc = PaymentService(should_fail=payment_fails)
        
        # Keep track of what succeeded so we know what to roll back
        self.compensation_stack = []

    def execute(self):
        print("=== Starting Trip Booking Saga ===")
        try:
            # Step 1: Book Flight
            if self.flight_svc.book():
                self.compensation_stack.append(self.flight_svc.cancel)

            # Step 2: Book Hotel
            if self.hotel_svc.book():
                self.compensation_stack.append(self.hotel_svc.cancel)

            # Step 3: Charge Payment
            if self.payment_svc.charge():
                # We don't add payment to compensation stack because 
                # if payment succeeds, the whole Saga is complete!
                pass

            print("✅ Saga Completed Successfully! Trip is booked.")
            
        except Exception as e:
            print(f"\n⚠️ Saga Failed at a step! Error: {e}")
            self.run_compensations()

    def run_compensations(self):
        print("🔄 Initiating Saga Rollback (Compensations)...")
        # Pop functions off the stack (LIFO order: Hotel then Flight)
        while self.compensation_stack:
            compensate_func = self.compensation_stack.pop()
            try:
                # In a real system, these MUST be retried until they succeed
                compensate_func()
            except Exception as e:
                print(f"CRITICAL: Compensation failed! {e}. Requires manual intervention.")
        
        print("🛑 Rollback Complete. System is consistent again.")


# ── Test Harness ──

if __name__ == "__main__":
    # Scenario 1: Happy Path
    print("--- Scenario 1: Successful Transaction ---")
    saga_success = TripBookingSaga(payment_fails=False)
    saga_success.execute()

    print("\n\n--- Scenario 2: Payment Fails ---")
    # Scenario 2: Payment fails, triggering rollbacks
    saga_fail = TripBookingSaga(payment_fails=True)
    saga_fail.execute()

Sample Output

--- Scenario 1: Successful Transaction ---
=== Starting Trip Booking Saga ===
  [Flight] ✈️  Booking flight...
  [Hotel] 🏨 Booking hotel room...
  [Payment] 💳 Charging credit card...
✅ Saga Completed Successfully! Trip is booked.


--- Scenario 2: Payment Fails ---
=== Starting Trip Booking Saga ===
  [Flight] ✈️  Booking flight...
  [Hotel] 🏨 Booking hotel room...
  [Payment] 💳 Charging credit card...

⚠️ Saga Failed at a step! Error: Insufficient Funds!
🔄 Initiating Saga Rollback (Compensations)...
  [Hotel] ❌ Cancelling hotel room (Compensation)...
  [Flight] ❌ Cancelling flight (Compensation)...
🛑 Rollback Complete. System is consistent again.

Notice the LIFO (Last-In-First-Out) order of compensations. Because the Payment failed, we first cancelled the Hotel, then cancelled the Flight, safely returning the system to its original state.

8. Complexity

9. Trade-offs

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. What is the Saga Pattern? Hint: A sequence of local transactions spanning multiple microservices. If one fails, compensating transactions are triggered to undo the preceding successful steps.

2. Why is Two-Phase Commit (2PC) rarely used in modern microservices? Hint: It is a blocking protocol. It requires taking out locks across multiple databases simultaneously, which ruins scalability. Furthermore, many modern NoSQL databases don't even support XA transactions.

3. What happens if a Compensating Transaction fails? Hint: The system must retry it indefinitely (Exponential Backoff). It must be idempotent. If it absolutely cannot succeed (e.g., database is permanently corrupted), the system must raise an alert for manual human intervention.

13. Used By (Downstream Blocks)

• 025 Event-Driven Architecture — Sagas are often implemented on top of Event-Driven messaging systems (Choreography).

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: Asynchronous Saga (Mock)

Modify the Python code to simulate an Asynchronous Orchestrator. Instead of calling flight_svc.book() directly, the Orchestrator should append a "COMMAND" to a mock message queue, and a separate worker thread should process the command and put a "REPLY" on a response queue.

Exercise 2: Semantic Locks

Modify the FlightService. Instead of book() and cancel(), implement reserve(saga_id) (sets state to PENDING), confirm(saga_id) (sets state to BOOKED), and release(saga_id) (deletes the reservation). The Orchestrator should call reserve during the Saga, and only call confirm at the very end when everything succeeds.

Website Metadata