DNS & Service Discovery

Metadata

1. Overview

What problem does it solve?

Computers communicate using IP addresses (192.168.1.5 or 2607:f8b0:4005:808::200e), but humans and code use names (google.com or payment-service).

• DNS (Domain Name System) solves this for the public internet, acting as the phonebook of the web.
• Service Discovery solves this for internal microservices, dynamically tracking which IP addresses belong to which services as containers spin up and down.

What breaks without it?

• Users would have to type IP addresses into their browsers.
• If a server's IP address changes (which happens constantly in cloud environments), clients would break.
• Microservices wouldn't know how to talk to each other in a dynamic environment like Kubernetes.
• Load balancers wouldn't know which backend instances are healthy and alive.

2. Motivation

Why was DNS invented?

In the early ARPANET days (1970s), every computer downloaded a single text file called HOSTS.TXT from the Stanford Research Institute. As the internet grew to thousands of computers, maintaining a centralized file became impossible. Paul Mockapetris invented DNS in 1983 as a hierarchical, distributed database to replace HOSTS.TXT.

Why was Service Discovery invented?

DNS is heavily cached and relatively slow to update. In modern microservices (especially with Docker and Kubernetes), instances of a payment-service might be created and destroyed every few minutes. Standard DNS caching (TTL) would route traffic to dead IPs. Internal Service Discovery (like Consul, Eureka, or ZooKeeper) was invented to provide real-time, strongly consistent mapping of service names to healthy IP addresses.

3. Real-World Usage

4. Prerequisites

5. Visual Explanation

The Public DNS Resolution Flow

When you type www.example.com:

sequenceDiagram
    participant C as Client (Browser)
    participant R as Recursive Resolver (ISP/8.8.8.8)
    participant Root as Root Server (.)
    participant TLD as TLD Server (.com)
    participant Auth as Auth Server (example.com)

    C->>R: 1. What is the IP for www.example.com?
    R->>Root: 2. Where is .com?
    Root-->>R: 3. Ask TLD Server at IP 1.2.3.4
    R->>TLD: 4. Where is example.com?
    TLD-->>R: 5. Ask Auth Server at IP 5.6.7.8
    R->>Auth: 6. What is the IP for www.example.com?
    Auth-->>R: 7. The IP is 93.184.216.34 (A Record)
    R-->>C: 8. The IP is 93.184.216.34 (Cached for 1 hour)

Internal Service Discovery (Client-Side)

sequenceDiagram
    participant API as API Gateway
    participant SR as Service Registry (Consul)
    participant P1 as Payment Service (IP: 10.0.0.1)
    participant P2 as Payment Service (IP: 10.0.0.2)

    P1->>SR: Register("payment-service", 10.0.0.1)
    P2->>SR: Register("payment-service", 10.0.0.2)
    
    Note over P1,SR: Services send heartbeats every 10s
    
    API->>SR: Lookup("payment-service")
    SR-->>API: [10.0.0.1, 10.0.0.2]
    
    API->>P1: HTTP POST /charge (Client-side load balancing)

6. Internal Working

6.1 DNS Record Types

DNS holds different types of records, not just IPs:

6.2 DNS TTL (Time To Live)

Every DNS record has a TTL (e.g., 3600 seconds).

• High TTL (24 hours): Less load on DNS servers, faster for users, but if your server IP changes, users will be broken for up to 24 hours.
• Low TTL (60 seconds): Great for fast failover and load balancing (DNS Round Robin), but higher latency as clients query DNS more often.

6.3 Service Discovery Mechanisms

Inside a datacenter, DNS caching is too slow. Service Discovery provides real-time state.

1. Service Registration: When a service boots up, it registers its IP and Port with the Registry. 2. Health Checking: The Registry (or the service) sends periodic heartbeats. If a service dies, it is evicted from the registry immediately. 3. Service Discovery:

• Client-side: The client asks the registry for IPs and picks one (e.g., Netflix Ribbon).
• Server-side: The client talks to a Load Balancer, and the Load Balancer queries the registry (e.g., AWS ALB, Nginx).

7. Implementation

Why Python? We will implement a simplified internal Service Registry over HTTP, showing exactly how microservices register themselves, send heartbeats, and how clients discover them.

"""
004 - Service Registry & Discovery
A simplified internal "Consul" or "Eureka".
Services register via POST, send heartbeats, and are automatically 
evicted if they die. Clients query via GET.
"""
import time
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
from urllib.parse import urlparse

# ── The Registry State ──

# Format: {"service_name": {"ip:port": last_heartbeat_timestamp}}
registry: dict[str, dict[str, float]] = {}
registry_lock = threading.Lock()

# How long before a service is considered dead (seconds)
TTL = 15 


# ── The Server ──

class ServiceRegistryHandler(BaseHTTPRequestHandler):
    def _send_json(self, status: int, data: dict):
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(data).encode("utf-8"))

    def do_POST(self):
        """Register or Heartbeat a service instance."""
        parsed_path = urlparse(self.path)
        
        # Path format: /register/<service_name>
        if parsed_path.path.startswith("/register/"):
            service_name = parsed_path.path.split("/")[-1]
            
            content_length = int(self.headers.get("Content-Length", 0))
            body = json.loads(self.rfile.read(content_length))
            instance_id = body.get("instance") # e.g. "10.0.0.1:8080"
            
            if not instance_id:
                self._send_json(400, {"error": "Missing 'instance' in body"})
                return

            with registry_lock:
                if service_name not in registry:
                    registry[service_name] = {}
                
                # Update the heartbeat timestamp
                registry[service_name][instance_id] = time.time()
                
            print(f"[REGISTER/HEARTBEAT] {service_name} -> {instance_id}")
            self._send_json(200, {"status": "registered", "ttl": TTL})
        else:
            self._send_json(404, {"error": "Not Found"})

    def do_GET(self):
        """Discover instances for a service."""
        parsed_path = urlparse(self.path)
        
        # Path format: /discover/<service_name>
        if parsed_path.path.startswith("/discover/"):
            service_name = parsed_path.path.split("/")[-1]
            
            with registry_lock:
                instances = registry.get(service_name, {})
                # Filter out dead instances on the fly
                now = time.time()
                healthy_instances = [
                    inst for inst, last_beat in instances.items()
                    if now - last_beat <= TTL
                ]
            
            self._send_json(200, {
                "service": service_name,
                "instances": healthy_instances
            })
        else:
            self._send_json(404, {"error": "Not Found"})

    # Suppress default HTTP logging to keep console clean
    def log_message(self, format, *args):
        pass


# ── Background Eviction Task ──

def evict_dead_services():
    """Periodically clean up instances that missed their heartbeats."""
    while True:
        time.sleep(5)
        now = time.time()
        with registry_lock:
            for service_name, instances in list(registry.items()):
                dead = [inst for inst, last_beat in instances.items() if now - last_beat > TTL]
                for d in dead:
                    del registry[service_name][d]
                    print(f"[EVICTED] {service_name} -> {d} (Missed heartbeat)")
                
                # Clean up empty services
                if not registry[service_name]:
                    del registry[service_name]


if __name__ == "__main__":
    # Start the eviction thread
    threading.Thread(target=evict_dead_services, daemon=True).start()

    port = 8500
    server = HTTPServer(("127.0.0.1", port), ServiceRegistryHandler)
    print(f"Service Registry running on port {port}...")
    print("Test via:")
    print("  curl -X POST http://127.0.0.1:8500/register/payment -d '{\"instance\":\"10.0.0.1:8080\"}'")
    print("  curl http://127.0.0.1:8500/discover/payment")
    server.serve_forever()

Sample Output

# Terminal 1: Run Registry
$ python registry.py
Service Registry running on port 8500...

# Terminal 2: Start Service Instance A
$ curl -X POST http://127.0.0.1:8500/register/payment \
    -d '{"instance":"10.0.0.1:8080"}'
{"status": "registered", "ttl": 15}

# Terminal 2: Start Service Instance B
$ curl -X POST http://127.0.0.1:8500/register/payment \
    -d '{"instance":"10.0.0.2:8080"}'

# Terminal 3: API Gateway queries discovery
$ curl http://127.0.0.1:8500/discover/payment
{"service": "payment", "instances": ["10.0.0.1:8080", "10.0.0.2:8080"]}

# Wait 15 seconds without sending heartbeats...
# Terminal 1 outputs:
[EVICTED] payment -> 10.0.0.1:8080 (Missed heartbeat)
[EVICTED] payment -> 10.0.0.2:8080 (Missed heartbeat)

# Terminal 3: Query again
$ curl http://127.0.0.1:8500/discover/payment
{"service": "payment", "instances": []}

8. Complexity

Scalability Characteristics

• DNS Scaling: Scales massively through hierarchical caching. The Root servers handle almost no traffic because your ISP and your OS cache the results.
• Service Registry Scaling: Usually deployed as a 3 or 5 node cluster using a consensus algorithm like Raft (Block 020). The clients maintain local caches of the registry state to reduce load on the cluster.

9. Trade-offs

Client-Side Discovery vs Server-Side Discovery

Modern trend: The Service Mesh (Block 034) extracts client-side discovery out of the application code and puts it into a sidecar proxy (Envoy), combining the best of both worlds.

10. Production Evolution

11. Common Bugs

12. Interview Questions

1. What happens if the Root DNS servers go down? Hint: Nothing immediately. TLDs and Auth servers are heavily cached by ISPs.

2. Why don't we use standard DNS for internal microservice discovery? Hint: DNS relies on TTL caching. In a dynamic environment, you can't afford to route traffic to a dead container for 60 seconds.

3. What is a CNAME record and why can't the root domain (example.com) be a CNAME? Hint: CNAME maps a name to a name. DNS RFCs forbid CNAMEs at the apex because it conflicts with NS and MX records.

4. In your service registry, how do you prevent a temporary network blip from causing a service to be evicted? Hint: Adjust the TTL, use multiple missed heartbeats as a threshold, or use active health checking.

13. Used By (Downstream Blocks)

• 007 Load Balancer — LBs use discovery to find backend IPs
• 012 CDN — CDNs use DNS CNAMEs and Anycast routing
• 027 Heartbeat & Failure Detection — Core mechanism of service registries
• 034 Service Mesh — Relies heavily on internal service registries

14. Used In (Case Studies)

15. Related Blocks

16. Try It Yourself

Exercise 1: Client-Side Load Balancer

Write a python script that acts as an API Gateway. It should call GET /discover/payment, parse the JSON, and then randomly select one of the instances from the list to "route" the request to.

Exercise 2: Active Health Checks

Instead of waiting for services to POST a heartbeat, modify the registry to maintain a list of /health endpoints for registered services. Have the eviction thread actively send an HTTP GET to those endpoints every 10 seconds.

Website Metadata