Heartbeat & Failure Detection

Metadata

1. Overview

What problem does it solve?

In a distributed system with hundreds of servers, servers will crash. Disks will fail, network cables will be unplugged, and kernel panics will occur.

When a server dies, the rest of the system needs to know about it quickly so it can:

1. Stop routing user traffic to the dead server.
2. Promote a replica to take over its database duties.
3. Spin up a replacement server to restore capacity.

A Heartbeat is a periodic signal generated by hardware or software to indicate normal operation. Failure Detection is the algorithm that interprets these heartbeats to decide if a node is truly dead.

What breaks without it?

• Blackholes: Load balancers keep sending traffic to a dead API server, causing 502 Bad Gateway errors for users.
• Data Unavailability: A primary database crashes, but because the cluster doesn't realize it's dead, no secondary database is promoted to take its place. The whole app goes down.

2. Motivation

You cannot simply wait for a TCP connection to return an error to know a server is dead. If a server loses power or a network switch dies, it won't send a TCP RST or FIN packet. Requests will just hang indefinitely until they hit a very long timeout (often minutes).

We need a proactive, fast way to detect death. Hence, servers must constantly yell "I'm alive!" every few seconds.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

The Ping-Ack Protocol (Pull-based)

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffcc00', 'edgeLabelBackground':'#ffffff'}}}%%
sequenceDiagram
    participant Monitor as Monitor Node (e.g. Load Balancer)
    participant NodeA as Application Node A
    
    Monitor->>NodeA: Ping (t=0s)
    NodeA-->>Monitor: Ack (I'm healthy!)
    
    Monitor->>NodeA: Ping (t=5s)
    NodeA-->>Monitor: Ack (I'm healthy!)
    
    Note over NodeA: Node A experiences Kernel Panic
    
    Monitor->>NodeA: Ping (t=10s)
    Note over Monitor: Waits 2 seconds for timeout...
    Monitor->>NodeA: Ping (t=12s) - Retry 1
    Note over Monitor: Waits 2 seconds for timeout...
    Monitor->>NodeA: Ping (t=14s) - Retry 2
    Note over Monitor: Waits 2 seconds for timeout...
    
    Note over Monitor: Node A marked DEAD (Evicted from Pool)

The Heartbeat Protocol (Push-based)

Instead of the monitor asking, the nodes proactively broadcast their status.

graph TD
    classDef monitor fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef node fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#155724;
    classDef dead fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#721c24;

    M((Registry / Monitor)):::monitor
    N1((Node 1)):::node
    N2((Node 2)):::node
    N3((Node 3)):::dead
    
    N1 -- "Push Heartbeat (t=10s)" --> M
    N2 -- "Push Heartbeat (t=10s)" --> M
    N3 -. "Silence (Last seen t=2s)" .-> M
    
    Note right of M: Monitor evicts Node 3 at t=12s

6. Internal Working

Types of Heartbeats

1. Pull (Active Health Checks)

   • The central authority (Load Balancer, API Gateway) pings the nodes.
   • Example: GET /health over HTTP.
   • Pros: Easy to implement.
   • Cons: Doesn't scale well to 10,000+ nodes (Monitor becomes a bottleneck).

2. Push (Passive Monitoring)

   • Nodes send UDP packets or HTTP requests to a central registry.
   • Pros: Monitor does less work.
   • Cons: If the network to the monitor is saturated, healthy nodes look dead.

3. Gossip Protocol (Peer-to-Peer)

   • No central monitor. Nodes randomly ping a few other nodes every second. If a node fails to respond, the news is gossiped to the rest of the cluster.
   • Pros: Infinitely scalable. No single point of failure.
   • Cons: Complex to implement. Detection takes slightly longer to propagate.

The Timeout Dilemma

How long should the Monitor wait before declaring a node dead?

• Too short (e.g., 500ms): A temporary network spike or a GC pause makes the node look dead. The system thrashes, constantly evicting and re-adding nodes (False Positives).
• Too long (e.g., 30s): User traffic is routed to a black hole for 30 seconds.

Solution: The \Phi (Phi) Accrual Failure Detector Instead of a strict binary (Alive/Dead), calculate the probability a node is dead based on historical response times. If a node usually responds in 10ms but currently hasn't responded in 2 seconds, the suspicion level (\Phi) is very high. This adapts to changing network conditions automatically. (Used by Cassandra).

7. Implementation

Why Go? Go's concurrency model (Goroutines + Tickers) is perfect for building continuous background monitoring loops that don't block the main application.

/*
027 - Heartbeat & Failure Detection
Simulates a central monitor (like a Load Balancer or ZooKeeper)
tracking the health of multiple nodes via Pull-based pinging.
*/
package main

import (
	"fmt"
	"sync"
	"time"
)

// Node Status constants
const (
	StatusHealthy = "HEALTHY"
	StatusSuspect = "SUSPECT" // Missed a ping, but not dead yet
	StatusDead    = "DEAD"
)

// Node represents a server being monitored
type Node struct {
	ID           string
	Address      string
	Status       string
	MissedPings  int
	MaxMisses    int
	IsActuallyUp bool // Simulates the real physical state
}

// Monitor tracks a cluster of nodes
type Monitor struct {
	Nodes map[string]*Node
	mu    sync.Mutex
}

func NewMonitor() *Monitor {
	return &Monitor{
		Nodes: make(map[string]*Node),
	}
}

func (m *Monitor) RegisterNode(id, address string) {
	m.mu.Lock()
	defer m.mu.Unlock()
	m.Nodes[id] = &Node{
		ID:           id,
		Address:      address,
		Status:       StatusHealthy,
		MissedPings:  0,
		MaxMisses:    3, // Require 3 consecutive misses to declare DEAD
		IsActuallyUp: true,
	}
	fmt.Printf("[Monitor] Registered new node: %s\n", id)
}

// Simulates sending an HTTP GET /health or a TCP Ping
func (m *Monitor) pingNode(node *Node) bool {
	// In reality, this would be an http.Get() with a tight timeout
	// Here we just read the simulated physical state
	return node.IsActuallyUp
}

// StartMonitoring begins the background heartbeat loop
func (m *Monitor) StartMonitoring(interval time.Duration) {
	ticker := time.NewTicker(interval)
	go func() {
		for range ticker.C {
			m.mu.Lock()
			for _, node := range m.Nodes {
				
				// Don't ping dead nodes endlessly in this simple implementation
				if node.Status == StatusDead {
					continue
				}

				success := m.pingNode(node)

				if success {
					if node.Status != StatusHealthy {
						fmt.Printf("[Monitor] 💚 Node %s recovered! Marking HEALTHY.\n", node.ID)
					}
					node.Status = StatusHealthy
					node.MissedPings = 0
				} else {
					node.MissedPings++
					node.Status = StatusSuspect
					fmt.Printf("[Monitor] ⚠️ Node %s missed ping (%d/%d)\n", node.ID, node.MissedPings, node.MaxMisses)

					if node.MissedPings >= node.MaxMisses {
						node.Status = StatusDead
						fmt.Printf("[Monitor] ❌ Node %s declared DEAD. Evicting from pool.\n", node.ID)
						// In real life: Trigger alert, remove from load balancer, initiate failover
					}
				}
			}
			m.mu.Unlock()
		}
	}()
}

// ── Test Harness ──

func main() {
	monitor := NewMonitor()
	
	monitor.RegisterNode("App-Server-1", "10.0.0.1:8080")
	monitor.RegisterNode("App-Server-2", "10.0.0.2:8080")

	// Ping every 500ms
	monitor.StartMonitoring(500 * time.Millisecond)

	time.Sleep(1 * time.Second)
	
	// Simulate Network Cable Unplugged on Server-2
	fmt.Println("\n--- Simulating Network Failure on App-Server-2 ---")
	monitor.mu.Lock()
	monitor.Nodes["App-Server-2"].IsActuallyUp = false
	monitor.mu.Unlock()

	// Wait for Failure Detector to notice and reach MaxMisses
	time.Sleep(2 * time.Second)

	// Simulate Recovery
	fmt.Println("\n--- Simulating Server-2 Reboot & Recovery ---")
	monitor.mu.Lock()
	monitor.Nodes["App-Server-2"].IsActuallyUp = true
	// Must reset status so monitor will ping it again in our naive implementation
	monitor.Nodes["App-Server-2"].Status = StatusSuspect 
	monitor.mu.Unlock()

	time.Sleep(1 * time.Second)
}

Sample Output

[Monitor] Registered new node: App-Server-1
[Monitor] Registered new node: App-Server-2

--- Simulating Network Failure on App-Server-2 ---
[Monitor] ⚠️ Node App-Server-2 missed ping (1/3)
[Monitor] ⚠️ Node App-Server-2 missed ping (2/3)
[Monitor] ⚠️ Node App-Server-2 missed ping (3/3)
[Monitor] ❌ Node App-Server-2 declared DEAD. Evicting from pool.

--- Simulating Server-2 Reboot & Recovery ---
[Monitor] 💚 Node App-Server-2 recovered! Marking HEALTHY.

8. Complexity

9. Trade-offs

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. How does a Load Balancer know when to stop sending traffic to a server? Hint: Active health checks (pinging /health every X seconds) and evicting after Y consecutive failures.

2. Why do we require 3 consecutive misses to declare a node dead instead of just 1? Hint: Networks are unreliable. Dropped packets are normal. A GC pause might delay a response. We need to prevent false positives and thrashing.

3. What is a "Zombie" node in failure detection? Hint: A node whose core business logic is deadlocked or broken, but its background health-check endpoint is still blindly returning HTTP 200 OK.

4. In a 10,000 node cluster, why might a centralized Pull-based monitor fail? Hint: The monitor has to open 10,000 TCP connections every 5 seconds. It runs out of sockets and CPU. Switch to Gossip.

13. Used By (Downstream Blocks)

• 022 Leader Election — You can't elect a new leader if you don't know the old one is dead.
• 028 Circuit Breaker — Uses localized failure detection (errors returned) to stop calling a dying downstream service.
• 033 Monitoring & Alerting — Heartbeats are the foundation of uptime monitoring.

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: \Phi Accrual (Simplified)

Modify the Go implementation to track the LastPingLatency. If the node responds in 10ms normally, but the latest ping takes 500ms, mark it SUSPECT immediately rather than waiting for a full missed ping timeout.

Exercise 2: Cascading Failure Protection

Add a check in the Monitor loop: If more than 50% of the nodes are currently marked SUSPECT or DEAD, log a massive warning "NETWORK PARTITION DETECTED" and temporarily pause evictions to prevent destroying the entire cluster due to a faulty switch.

Website Metadata