Monitoring & Alerting

Metadata

1. Overview

What problem does it solve?

If you launch an e-commerce website and go to sleep, how do you know if it's actually working? If the database runs out of disk space, how do you know? If a bad code deployment causes checkout errors to spike from 1% to 15%, who notices?

Monitoring is the process of collecting quantitative data (Metrics) about your system (e.g., CPU usage, HTTP requests per second, error rates) in real-time. Alerting is the automated process of paging a human (e.g., via PagerDuty or Slack) when those metrics cross a dangerous threshold.

What breaks without it?

• Silent Outages: Your users become your monitoring system. You only find out your site is down when angry customers yell at you on Twitter.
• Resource Exhaustion: A memory leak slowly consumes RAM over 3 days until the server crashes. With monitoring, you see the trend line going up and fix it on day 1.

2. Motivation

In the early days, monitoring meant writing a shell script that pinged a server every 5 minutes and emailed you if it didn't respond (e.g., Nagios).

As systems moved to microservices and Kubernetes, servers became ephemeral. You couldn't just ping "Server A" because Server A might be destroyed and replaced by Server B dynamically based on load. The industry needed a dynamic, time-series based monitoring system. Prometheus (created at SoundCloud, inspired by Google's internal 'Borgmon') became the open-source standard for cloud-native metrics collection.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

The Push vs Pull Model

There are two ways to collect metrics:

1. Push Model (e.g., StatsD / Datadog): The application actively sends UDP packets containing metrics to a centralized server. Problem: If the metric server goes down, the application might block or lose data.

2. Pull Model (e.g., Prometheus): The application simply maintains a /metrics HTTP endpoint. Prometheus runs a background loop that scrapes (HTTP GET) that endpoint every 15 seconds. Benefit: If Prometheus goes down, the application doesn't care. It just keeps serving its /metrics endpoint.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffcc00', 'edgeLabelBackground':'#ffffff'}}}%%
graph TD
    classDef app fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef prom fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef dash fill:#cce5ff,stroke:#007bff,stroke-width:2px;

    subgraph Microservices
        A[Order Service<br/>:8080/metrics]:::app
        B[Payment Service<br/>:8081/metrics]:::app
    end

    P[(Prometheus TSDB)]:::prom -->|"Scrapes every 15s"| A
    P -->|"Scrapes every 15s"| B
    
    G[Grafana Dashboards]:::dash -->|"Query (PromQL)"| P
    
    Alert[Alertmanager]:::prom -->|"Error Rate > 5%"| Pager[PagerDuty / Slack]
    P -->|"Fires Alert"| Alert

6. Internal Working

Types of Metrics

1. Counter: A number that only goes up (or resets to 0 on restart).
   • E.g., http_requests_total.
   • You use math (rate()) in Grafana to figure out the Requests Per Second.
2. Gauge: A number that goes up and down.
   • E.g., active_connections, cpu_usage_percent, queue_depth.
3. Histogram: Samples observations into "Buckets" to calculate percentiles (p99, p95).
   • E.g., http_request_duration_seconds. If you want to know "99% of my users experience a page load faster than 200ms", you use a Histogram.

The Four Golden Signals

Google's SRE (Site Reliability Engineering) handbook states that if you can only measure four things, measure these:

1. Latency: How long requests take (e.g., p99 latency is 300ms).
2. Traffic: How much demand is on the system (e.g., 500 requests per second).
3. Errors: The rate of failed requests (e.g., 2% of requests return HTTP 500).
4. Saturation: How "full" your system is (e.g., CPU is at 90%, Database connection pool is 100% full).

7. Implementation

Why Go? Go is the language Prometheus is written in. It has the best, most native support for exposing a /metrics endpoint using the official prometheus/client_golang library.

/*
033 - Monitoring & Alerting
Demonstrates how to instrument a web server to expose
Prometheus metrics (Counters, Gauges, and Histograms).
*/
package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

// ── 1. Define Metrics ──

var (
	// COUNTER: Only goes up. Good for total requests.
	requestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total number of HTTP requests processed",
		},
		[]string{"path", "status"}, // Labels (tags) to slice the data
	)

	// GAUGE: Goes up and down. Good for current state.
	activeRequests = prometheus.NewGauge(
		prometheus.GaugeOpts{
			Name: "http_active_requests",
			Help: "Number of requests currently being processed",
		},
	)

	// HISTOGRAM: Buckets data. Good for latency percentiles (p99).
	requestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "http_request_duration_seconds",
			Help:    "Latency of HTTP requests",
			Buckets: []float64{0.1, 0.5, 1.0, 2.0, 5.0}, // Define buckets in seconds
		},
		[]string{"path"},
	)
)

func init() {
	// Register metrics with Prometheus's default registry
	prometheus.MustRegister(requestsTotal)
	prometheus.MustRegister(activeRequests)
	prometheus.MustRegister(requestDuration)
}

// ── 2. The Application Logic ──

func checkoutHandler(w http.ResponseWriter, r *http.Request) {
	// 1. Increment Active Requests Gauge
	activeRequests.Inc()
	defer activeRequests.Dec()

	// 2. Start a timer for the Histogram
	start := time.Now()
	
	// Simulate work (Random sleep between 0 and 2 seconds)
	sleepTime := time.Duration(rand.Intn(2000)) * time.Millisecond
	time.Sleep(sleepTime)

	// Simulate occasional 500 Errors
	status := "200"
	if rand.Float32() < 0.2 { // 20% failure rate
		status = "500"
		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
	} else {
		fmt.Fprintf(w, "Checkout Successful!")
	}

	// 3. Record Duration (Histogram)
	duration := time.Since(start).Seconds()
	requestDuration.WithLabelValues("/checkout").Observe(duration)

	// 4. Record Total Requests (Counter)
	requestsTotal.WithLabelValues("/checkout", status).Inc()
}


// ── 3. Test Harness ──

func main() {
	// Route 1: The actual application logic
	http.HandleFunc("/checkout", checkoutHandler)
	
	// Route 2: The Prometheus Metrics Exporter (Pull Model)
	http.Handle("/metrics", promhttp.Handler())

	fmt.Println("🚀 Server running on :8080")
	fmt.Println("👉 Hit http://localhost:8080/checkout to generate traffic")
	fmt.Println("📈 View metrics at http://localhost:8080/metrics")

	// Start server
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Sample Output (from /metrics)

If you visit http://localhost:8080/metrics after hitting the checkout endpoint a few times, you'll see the raw data Prometheus scrapes:

# HELP http_active_requests Number of requests currently being processed
# TYPE http_active_requests gauge
http_active_requests 0

# HELP http_request_duration_seconds Latency of HTTP requests
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{path="/checkout",le="0.1"} 0
http_request_duration_seconds_bucket{path="/checkout",le="0.5"} 1
http_request_duration_seconds_bucket{path="/checkout",le="1"} 3
http_request_duration_seconds_bucket{path="/checkout",le="2"} 5
http_request_duration_seconds_bucket{path="/checkout",le="+Inf"} 5
http_request_duration_seconds_sum{path="/checkout"} 5.432
http_request_duration_seconds_count{path="/checkout"} 5

# HELP http_requests_total Total number of HTTP requests processed
# TYPE http_requests_total counter
http_requests_total{path="/checkout",status="200"} 4
http_requests_total{path="/checkout",status="500"} 1

Prometheus reads this text, stores it, and Grafana translates it into beautiful line charts.

8. Complexity

9. Trade-offs

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. What is the difference between Logs and Metrics? Hint: Logs are discrete events (e.g., User 5 bought Item X). Metrics are aggregations over time (e.g., The system processed 500 requests per second with a 2% error rate).

2. Why shouldn't you put user_id as a label on a Prometheus metric? Hint: Cardinality explosion. Prometheus stores a unique time series for every unique combination of labels. Millions of users = millions of series in RAM, which will crash the TSDB.

3. What are the Four Golden Signals? Hint: Latency, Traffic, Errors, and Saturation. If you monitor these four, you have a solid grasp on the health of any system.

4. Explain the difference between a Push model (StatsD) and a Pull model (Prometheus). Hint: Push sends data to a central server; Pull exposes an endpoint for the central server to scrape. Pull is generally preferred in cloud-native environments because it's easier to scale and avoids overwhelming the app if the metrics server goes down.

13. Used By (Downstream Blocks)

• 010 Rate Limiter — Rate limiters rely heavily on monitoring to visualize how much traffic is being dropped.
• 028 Circuit Breaker — Circuit breaker state changes (OPEN/CLOSED) must be exported as metrics to trigger alerts.

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: Gauge vs Counter

Add a new metric: a Counter tracking payments_failed_total. Then add a Gauge tracking inventory_items_remaining (start it at 100). When a checkout succeeds, decrement the gauge. Notice how the Counter only goes up, but the Gauge goes down.

Exercise 2: PromQL Basics

If you have Docker installed, run a local Prometheus container pointing to your Go app. Write a PromQL query to calculate the percentage of errors over the last 1 minute: rate(http_requests_total{status="500"}[1m]) / sum(rate(http_requests_total[1m]))

Website Metadata