latency-optimization

What this skill does

# Latency Optimization

Comprehensive guide to reducing end-to-end latency in distributed systems - from network to application to database layers.

## When to Use This Skill

- Optimizing response times for user-facing applications
- Creating latency budgets for distributed systems
- Implementing geographic routing strategies
- Reducing database query latency
- Optimizing API response times
- Understanding and measuring latency components

## Latency Fundamentals

### Understanding Latency

```text
Latency Components:

Total Latency = Network + Processing + Queue + Serialization

┌─────────────────────────────────────────────────────────────┐
│                     Request Journey                          │
│                                                              │
│  Client ──► DNS ──► TCP ──► TLS ──► Server ──► DB ──► Back  │
│                                                              │
│  Components:                                                 │
│  ├── DNS Resolution: 0-100ms (cached: 0ms)                  │
│  ├── TCP Handshake: 1 RTT (~10-200ms)                       │
│  ├── TLS Handshake: 1-2 RTT (~20-400ms)                     │
│  ├── Request Transfer: depends on size                       │
│  ├── Server Processing: application-specific                 │
│  ├── Database Query: 1-1000ms typical                       │
│  └── Response Transfer: depends on size                      │
└─────────────────────────────────────────────────────────────┘

Key Metrics:
- P50: Median latency (50th percentile)
- P95: 95th percentile (tail latency starts)
- P99: 99th percentile (important for SLOs)
- P99.9: Three nines (critical systems)
```

### Latency Numbers Every Developer Should Know

```text
Latency Reference (2024 estimates):

Operation                              Time
─────────────────────────────────────────────────────
L1 cache reference                     1 ns
L2 cache reference                     4 ns
Branch mispredict                      5 ns
L3 cache reference                     10 ns
Mutex lock/unlock                      25 ns
Main memory reference                  100 ns
Compress 1KB with Snappy              2,000 ns (2 μs)
SSD random read                       16,000 ns (16 μs)
Read 1 MB from memory                 50,000 ns (50 μs)
Read 1 MB from SSD                    200,000 ns (200 μs)
Round trip same datacenter            500,000 ns (500 μs)
Read 1 MB from network (1Gbps)        10,000,000 ns (10 ms)
HDD random read                       10,000,000 ns (10 ms)
Round trip US East to US West         40,000,000 ns (40 ms)
Round trip US to Europe               80,000,000 ns (80 ms)
Round trip US to Asia                 150,000,000 ns (150 ms)

Key Insights:
- Memory is 100x faster than SSD
- Same-datacenter is 80x faster than cross-continent
- Caching at any level provides huge wins
```

### Latency Budget

```text
Latency Budget Example (200ms target):

┌─────────────────────────────────────────────────────────────┐
│                    200ms Total Budget                        │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────┬──────────┬──────────┬──────────┬──────────┐  │
│  │ Network  │   Auth   │  Service │    DB    │ Response │  │
│  │   50ms   │   20ms   │   50ms   │   60ms   │   20ms   │  │
│  └──────────┴──────────┴──────────┴──────────┴──────────┘  │
│                                                              │
│  Breakdown:                                                  │
│  ├── Network (client → edge → origin): 50ms                 │
│  ├── Authentication/Authorization: 20ms                      │
│  ├── Service Processing: 50ms                               │
│  ├── Database Queries: 60ms                                 │
│  └── Response Serialization + Transfer: 20ms                │
└─────────────────────────────────────────────────────────────┘

Budget Rules:
1. Allocate budgets based on criticality
2. Leave 10-20% headroom for variance
3. Monitor P99 against budget
4. Alert when consistently over budget
5. Renegotiate budgets as system evolves
```

## Network Latency Optimization

### Geographic Routing

```text
Geographic Routing Strategies:

1. GeoDNS Routing
   User IP ──► DNS Resolver ──► Nearest Server IP

   Pros: Simple, works everywhere
   Cons: DNS caching, IP geolocation inaccuracy

2. Anycast Routing
   Same IP advertised from multiple locations
   BGP routes to nearest (network topology)

   Pros: Instant failover, no DNS delay
   Cons: Requires BGP expertise, stateful sessions tricky

3. Load Balancer Geo-routing
   Global LB ──► Regional LB ──► Servers

   Pros: Fine-grained control, health checking
   Cons: Adds latency hop, more complex

Selection Guide:
┌──────────────────┬─────────────────────────────────────┐
│ Use Case         │ Recommended Approach                │
├──────────────────┼─────────────────────────────────────┤
│ Static content   │ Anycast CDN                         │
│ API services     │ GeoDNS + Regional deployments       │
│ Real-time apps   │ Anycast + Connection persistence    │
│ Stateful apps    │ GeoDNS with session affinity        │
└──────────────────┴─────────────────────────────────────┘
```

### Protocol Optimization

```text
Protocol-Level Optimizations:

1. HTTP/2 Benefits
   ├── Multiplexing (no head-of-line blocking)
   ├── Header compression (HPACK)
   ├── Server push (preemptive responses)
   └── Single connection (reduced handshakes)

   Latency Impact: 20-50% improvement typical

2. HTTP/3 (QUIC) Benefits
   ├── 0-RTT connection resumption
   ├── No TCP head-of-line blocking
   ├── Built-in encryption
   └── Connection migration (IP changes)

   Latency Impact: 10-30% over HTTP/2

3. TLS Optimization
   ├── TLS 1.3 (1-RTT handshake)
   ├── Session resumption (0-RTT)
   ├── OCSP stapling (no CA roundtrip)
   └── Certificate chain optimization

   Latency Impact: 50-200ms saved per connection

4. TCP Optimization
   ├── TCP Fast Open (TFO)
   ├── Increased initial congestion window
   ├── BBR congestion control
   └── Keep-alive for connection reuse
```

### Connection Optimization

```text
Connection Strategies:

1. Connection Pooling
   ┌─────────────────────────────────────────┐
   │           Connection Pool               │
   │  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐      │
   │  │Conn1│ │Conn2│ │Conn3│ │Conn4│      │
   │  └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘      │
   └─────┼──────┼──────┼──────┼────────────┘
         │      │      │      │
      Reuse connections, avoid handshake cost

2. Preconnect/Prefetch
   <link rel="preconnect" href="https://api.example.com">
   <link rel="dns-prefetch" href="https://cdn.example.com">

   Triggers early connection establishment

3. Connection Coalescing (HTTP/2)
   Multiple domains → single connection
   (When sharing same IP and certificate)
```

## Application Latency Optimization

### Caching Strategies

```text
Caching Layers:

┌─────────────────────────────────────────────────────────────┐
│                    Caching Hierarchy                         │
│                                                              │
│  Browser ──► CDN Edge ──► App Cache ──► DB Cache ──► DB     │
│    1ms        10ms          20ms         50ms       100ms   │
│                                                              │
│  Each layer should catch most requests before next layer    │
└─────────────────────────────────────────────────────────────┘

Cache Type Selection:
┌──────────────────┬─────────────────┬────────────────────────┐
│ Data Type        │ Cache Location  │ TTL Strategy           │
├──────────────────┼─────────────────┼────────────────────────┤
│ Static assets    │ CDN + Browser   │ Long (1 year), hashed  │
│ API responses    │ CDN + App       │ Short (seconds-mins)   │
│ Session data     │ App (Redis)     │ Session duration       │
│ DB query results │ App (local/dist)│ Varies by query        │
│ Computed res