WhatsApp (Chat Application)

Metadata

Building Blocks Used

This case study builds upon the following foundational concepts:

• WebSockets & Long Polling (Persistent connections)
• Message Queues (Asynchronous message routing)
• Database Replication (High availability for user data)

1. System Requirements

Functional Requirements

• Users can send and receive text messages in real-time.
• One-on-one chatting (MVP) and Group chatting.
• Read receipts (Sent, Delivered, Read).
• Push notifications when the user is offline.
• Messages are End-to-End Encrypted (E2EE).

Non-Functional Requirements

• Extremely Low Latency: Messages must be delivered in under 200ms.
• High Availability: Users expect chat to work 24/7 without fail.
• Massive Connection Scale: Must support hundreds of millions of concurrently open connections.

2. Back-of-the-Envelope Estimation

• DAU (Daily Active Users): 1 Billion.
• Messages per user per day: 50.
• Total Messages per day: 50 Billion.

QPS (Queries Per Second)

• 50 Billion / (24 × 3600) ≈ 600,000 Messages / second.
• Peak QPS might be 2x or 3x this during holidays (e.g., New Year's Eve).
• Takeaway: A standard REST API with polling will completely collapse under this load. We must use a persistent connection protocol.

Storage Estimation (Per Year)

Assuming messages are only stored temporarily until delivered (like original WhatsApp) or stored permanently (like Telegram/Messenger): Let's assume permanent storage for this design, 100 bytes per message.

• 50 Billion × 100 bytes × 365 days = ~1.8 Petabytes / year.
• Takeaway: We need a massive, highly scalable NoSQL database tailored for write-heavy workloads (e.g., Cassandra).

3. High-Level Architecture

Traditional HTTP Request-Response doesn't work for chat. If Alice wants to send a message to Bob, Bob cannot wait for his phone to "refresh" the page. The server must push the message to Bob immediately. We solve this using WebSockets.

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#ffcc00', 'edgeLabelBackground':'#ffffff'}}}%%
graph TD
    classDef client fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef conn fill:#cce5ff,stroke:#007bff,stroke-width:2px;
    classDef mq fill:#ffeeba,stroke:#ffc107,stroke-width:2px;
    classDef db fill:#f8d7da,stroke:#dc3545,stroke-width:2px;
    classDef svc fill:#d4edda,stroke:#28a745,stroke-width:2px;

    Alice((Alice)):::client <-->|WebSocket| C1[Connection Server 1]:::conn
    Bob((Bob)):::client <-->|WebSocket| C2[Connection Server 2]:::conn
    
    C1 -->|Publish| MQ[(Message Queue<br/>Kafka)]:::mq
    MQ -->|Consume| MS[Message Service]:::svc
    
    MS -->|Lookup Bob's Server| Redis[(Session Cache)]:::db
    MS -->|Push to C2| C2
    MS -->|Persist| DB[(Cassandra)]:::db

4. Deep Dive: Connection Servers

A single Linux server can handle a maximum of 65,535 TCP ports... if it is connecting to an external IP. However, when accepting incoming connections, a server can handle millions of concurrent WebSockets, bound only by RAM and File Descriptors.

To support 1 Billion concurrent users, we need thousands of Connection Servers. Their ONLY job is to hold open WebSocket connections. They contain zero business logic.

Session Management

When Bob connects to Connection Server 2 (CS2), CS2 must tell the rest of the backend: "Hey, Bob is connected to me!" It writes this into a Redis Session Cache: { "user_id": "Bob123", "server_id": "CS2" }

5. Deep Dive: Message Flow (Alice to Bob)

1. Alice sends a message to Bob over her WebSocket to CS1.
2. CS1 places the message onto a Message Queue (Kafka).
   • Why a Queue? If Alice goes offline immediately after hitting send, the backend still has the message safely durably stored on disk in Kafka. It acts as a shock absorber during New Year's Eve traffic spikes.
3. The Message Service consumes the message from Kafka.
4. It queries the Redis Session Cache: "Where is Bob?"
5. Redis replies: "Bob is on CS2."
6. The Message Service sends an RPC call (or uses another MQ topic) to CS2 containing the message.
7. CS2 pushes the message down the open WebSocket directly to Bob.

What if Bob is offline?

At Step 5, Redis replies: "Bob is not connected." The Message Service instead calls Apple Push Notification Service (APNS) or Google Firebase (FCM) to send a wake-up push notification to Bob's phone.

6. Deep Dive: Database Design

Chat history is incredibly write-heavy and append-only. Users almost never edit or delete old messages. Furthermore, users only read the most recent messages; old messages are rarely accessed.

This perfectly matches the architecture of an LSM Tree database like Cassandra.

Table: messages

Because we use chat_id as the Partition Key (see Database Sharding), loading a chat's history is a blindingly fast Sequential I/O read on a single disk.

7. Deep Dive: End-to-End Encryption (E2EE)

In modern chat apps, the server cannot read the messages.

When Alice wants to message Bob:

1. Alice's phone requests Bob's Public Key from the server.
2. Alice's phone encrypts the message using Bob's Public Key.
3. The ciphertext travels through the WebSockets, Kafka, and Cassandra. The server only sees gibberish.
4. Bob receives the ciphertext and decrypts it locally on his phone using his Private Key (which never leaves his device).

Note: In reality, WhatsApp uses the Signal Protocol, which rotates keys constantly for Perfect Forward Secrecy.

8. Bottlenecks & Trade-offs

The "Thundering Herd" Problem (Group Chats)

If 100,000 people are in a massive group chat, and one person sends a message, the Message Service must do 100,000 lookups in Redis to find 100,000 Connection Servers, and then fan-out 100,000 pushes. This will instantly melt the system. Solution: For massive group chats, we reverse the model. We don't push to everyone. Instead, users are assigned to a Pub/Sub topic. We drop the message in a dedicated Kafka partition, and the Connection Servers handling those users "pull" the message.

Presence (Online Status)

Updating the "Online" status indicator every time a user switches apps would generate an astronomical amount of traffic (far more than actual messages). Solution: Send presence heartbeats infrequently (e.g., every 30 seconds), and only broadcast a user's presence changes to their active chats, not their entire contact list.

9. Summary of Building Blocks Used

This case study brings together the following fundamental components:

Website Metadata