Load Balancer

Metadata

1. Overview

What problem does it solve?

A single server can only handle so much traffic before it runs out of CPU, memory, or network bandwidth. To handle more traffic, you must scale horizontally by adding more servers.

But if you have 10 identical API servers, how does the client know which one to talk to? A Load Balancer (LB) sits in front of your servers, acts as the single point of contact for clients, and distributes incoming requests evenly across the backend fleet.

What breaks without it?

• Availability: If a server crashes, clients talking to that server will fail. A load balancer detects the crash and routes traffic away from it.
• Scalability: You are hard-capped by the limits of a single machine.
• Security: Backend servers are exposed directly to the public internet.

2. Motivation

The Evolution of Load Balancing

1. DNS Round Robin (The 90s): Map one domain (example.com) to multiple IP addresses in DNS. Problem: DNS caches meant clients kept hitting dead IPs for hours after a server died.
2. Hardware Load Balancers (The 2000s): Expensive, proprietary physical appliances (F5 Networks, Citrix) sitting in data centers. Extremely fast, but inflexible.
3. Software Load Balancers (The 2010s): HAProxy, Nginx. Software running on commodity Linux boxes. Cheaper and programmable.
4. Cloud & Proxy Era (Present): AWS ALB/NLB, Envoy Proxy. Load balancers are now distributed systems themselves, capable of advanced L7 routing, rate limiting, and observability.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

L4 vs L7 Load Balancing

graph TD
    Client["Client"] --> LB{"Load Balancer"}
    
    subgraph "Layer 4 (Transport)"
        LB -- "TCP Port 80" --> S1["Server 1"]
        LB -- "TCP Port 80" --> S2["Server 2"]
    end
    
    subgraph "Layer 7 (Application)"
        LB -- "GET /images/*" --> S3["Image Servers"]
        LB -- "POST /api/*" --> S4["API Servers"]
    end

High Availability Architecture (Redundant LBs)

If a load balancer is the single point of entry, what happens if the LB crashes?

graph TD
    DNS["DNS (example.com)"] --> |Active IP| LB1["Primary LB"]
    DNS -.-> |Standby IP| LB2["Secondary LB"]
    
    LB1 --> S1["Server A"]
    LB1 --> S2["Server B"]
    
    LB2 -.-> S1
    LB2 -.-> S2
    
    LB1 <-.-> |Heartbeat (Keepalived/VRRP)| LB2

6. Internal Working

6.1 Layer 4 (Network) vs Layer 7 (Application)

6.2 Routing Algorithms

When a request arrives, how does the LB choose which server gets it?

1. Round Robin: Server 1, Server 2, Server 3, Server 1, Server 2... (Simple, but ignores server load).
2. Weighted Round Robin: Server 1 gets 2x more traffic than Server 2 (Useful when servers have different CPU capacities).
3. Least Connections: Send request to the server with the fewest active connections (Best for long-lived requests like WebSockets).
4. IP Hash: hash(Client IP) % N. Ensures the same client always hits the same server (Sticky Sessions).

6.3 Active Health Checks

A load balancer is useless if it sends traffic to a dead server. The LB constantly pings backend servers (e.g., GET /health every 5 seconds). If a server fails 3 consecutive checks, it is removed from the routing pool. When it passes the check again, it is added back.

7. Implementation

Why Go? Go is the undisputed king of modern network proxies (Traefik, CoreDNS, Caddy are all written in Go). Its built-in ReverseProxy struct and ultra-efficient goroutines make it trivial to build a high-performance L7 load balancer.

/*
007 - L7 HTTP Load Balancer
A concurrent load balancer with Round Robin routing, 
Active Health Checks, and HTTP Reverse Proxying.
Run: `go run loadbalancer.go`
*/
package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
	"sync"
	"time"
)

// ── Backend Server Representation ──

type Backend struct {
	URL          *url.URL
	Alive        bool
	mux          sync.RWMutex
	ReverseProxy *httputil.ReverseProxy
}

func (b *Backend) SetAlive(alive bool) {
	b.mux.Lock()
	b.Alive = alive
	b.mux.Unlock()
}

func (b *Backend) IsAlive() bool {
	b.mux.RLock()
	alive := b.Alive
	b.mux.RUnlock()
	return alive
}

// ── Server Pool (Manages all backends) ──

type ServerPool struct {
	backends []*Backend
	current  uint64 // For Round Robin
}

// NextIndex atomically gets the next backend index (Round Robin)
func (s *ServerPool) NextIndex() int {
	// A real implementation would use atomic.AddUint64 for thread safety
	s.current++
	return int(s.current % uint64(len(s.backends)))
}

// GetNextPeer returns the next available healthy server
func (s *ServerPool) GetNextPeer() *Backend {
	// We loop through the backends to find a healthy one
	next := s.NextIndex()
	l := len(s.backends) + next // start from next and wrap around
	
	for i := next; i < l; i++ {
		idx := i % len(s.backends)
		if s.backends[idx].IsAlive() {
			s.current = uint64(idx)
			return s.backends[idx]
		}
	}
	return nil // All servers are dead!
}

// ── Active Health Checking ──

func (s *ServerPool) HealthCheck() {
	for {
		for _, b := range s.backends {
			status := "up"
			// Send a quick HTTP GET to the backend
			resp, err := http.Get(b.URL.String())
			
			if err != nil || resp.StatusCode != http.StatusOK {
				b.SetAlive(false)
				status = "down"
			} else {
				b.SetAlive(true)
			}
			
			if resp != nil {
				resp.Body.Close()
			}
			log.Printf("HealthCheck: %s is [%s]\n", b.URL, status)
		}
		time.Sleep(10 * time.Second)
	}
}

// ── Load Balancer Entry Point ──

func lbHandler(w http.ResponseWriter, r *http.Request) {
	peer := serverPool.GetNextPeer()
	if peer != nil {
		log.Printf("Routing '%s' to -> %s", r.URL.Path, peer.URL)
		// Transparently forward the request
		peer.ReverseProxy.ServeHTTP(w, r)
		return
	}
	http.Error(w, "Service Unavailable: No healthy backends", http.StatusServiceUnavailable)
}

var serverPool ServerPool

func main() {
	// Initialize three mock backend URLs
	servers := []string{
		"http://127.0.0.1:8081",
		"http://127.0.0.1:8082",
		"http://127.0.0.1:8083",
	}

	for _, s := range servers {
		serverUrl, err := url.Parse(s)
		if err != nil {
			log.Fatal(err)
		}
		
		proxy := httputil.NewSingleHostReverseProxy(serverUrl)
		
		serverPool.backends = append(serverPool.backends, &Backend{
			URL:          serverUrl,
			Alive:        true,
			ReverseProxy: proxy,
		})
	}

	// Start health checking in the background
	go serverPool.HealthCheck()

	log.Println("Load Balancer listening on port 8080...")
	server := http.Server{
		Addr:    ":8080",
		Handler: http.HandlerFunc(lbHandler),
	}
	
	// Start the load balancer
	if err := server.ListenAndServe(); err != nil {
		log.Fatal(err)
	}
}

Note: To fully test this, you would need to run three simple HTTP servers on ports 8081, 8082, and 8083.

8. Complexity

Scalability Characteristics

If a single Load Balancer caps out at 100k RPS, how do massive companies handle 1M+ RPS? Tiered Load Balancing:

1. DNS uses Geo-Routing to send you to a specific datacenter.
2. An L4 Hardware LB (or ECMP router) distributes TCP connections to a fleet of L7 Software LBs.
3. The L7 Software LBs inspect the HTTP headers and route to the actual application servers.

9. Trade-offs

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. What is the difference between an L4 and L7 load balancer? Hint: L4 only sees IPs and Ports (TCP/UDP). L7 decrypts traffic and sees HTTP URLs, headers, and cookies.

2. How does a load balancer prevent a single point of failure? Hint: You deploy them in Active-Passive pairs using a protocol like VRRP. If the primary dies, the secondary instantly takes over the IP address.

3. Why is Round Robin a bad idea for a service where request processing times vary wildly (10ms to 10s)? Hint: Long requests pile up on one server by chance, while other servers sit idle. Use Least Connections instead.

4. Your application relies on users hitting the exact same server so they don't get logged out. How do you configure the LB? Hint: IP Hash routing (Sticky Sessions). (Follow-up: This is a bad architecture. State should be moved to Redis so LBs can route anywhere).

13. Used By (Downstream Blocks)

• 008 API Gateway — Often combined with or placed directly behind an L7 load balancer.
• 016 Consistent Hashing — The advanced routing algorithm used by LBs in distributed databases.
• 034 Service Mesh — Envoy acts as a local L7 load balancer running next to every single microservice.

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: Least Connections Routing

Modify the Go implementation. Add an ActiveConnections int32 field to the Backend struct. Increment it before ServeHTTP and decrement it after. Change GetNextPeer() to return the healthy backend with the lowest ActiveConnections instead of Round Robin.

Exercise 2: Add X-Forwarded-For

In the lbHandler, before calling ServeHTTP, modify the r.Header to inject the X-Forwarded-For header using the client's IP address (extracted from r.RemoteAddr).

Website Metadata