observability-patterns

What this skill does

# Observability Patterns

Patterns for implementing comprehensive observability including logs, metrics, traces, and their correlation.

## When to Use This Skill

- Designing observability strategy
- Implementing the three pillars
- Correlating signals across systems
- Choosing observability tools
- Building monitoring dashboards

## What is Observability?

```text
Observability = Ability to understand internal state
                from external outputs

Not just monitoring (known-unknowns)
But understanding (unknown-unknowns)

Traditional monitoring: "Is CPU > 80%?"
Observability: "Why are users experiencing latency?"
```

## The Three Pillars

### Overview

```text
┌─────────────────────────────────────────────────────────┐
│                    OBSERVABILITY                         │
│                                                          │
│   ┌──────────┐    ┌──────────┐    ┌──────────┐         │
│   │   LOGS   │    │ METRICS  │    │  TRACES  │         │
│   │          │    │          │    │          │         │
│   │ Events   │    │ Counters │    │ Requests │         │
│   │ Details  │    │ Gauges   │    │ Spans    │         │
│   │ Context  │    │ Trends   │    │ Flow     │         │
│   └──────────┘    └──────────┘    └──────────┘         │
│        │               │               │                │
│        └───────────────┼───────────────┘                │
│                        │                                │
│               ┌────────┴────────┐                       │
│               │   CORRELATION   │                       │
│               │  (trace_id)     │                       │
│               └─────────────────┘                       │
└─────────────────────────────────────────────────────────┘

Each pillar answers different questions:
- Logs: What happened? (events)
- Metrics: How much/many? (aggregates)
- Traces: Where? (request flow)
```

### Logs

```text
Purpose: Discrete events with context

Structure:
{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "level": "ERROR",
  "service": "order-service",
  "message": "Payment failed",
  "trace_id": "abc123",
  "span_id": "def456",
  "user_id": "12345",
  "order_id": "ORD-789",
  "error": {
    "code": "CARD_DECLINED",
    "message": "Insufficient funds"
  }
}

Best for:
- Debugging specific issues
- Audit trails
- Error details
- Business events

Challenges:
- High volume → storage costs
- Unstructured → hard to query
- No aggregation → not for trends
```

### Metrics

```text
Purpose: Numeric measurements over time

Types:
┌─────────────────────────────────────────────────────────┐
│ Counter: Cumulative, only increases                     │
│ - http_requests_total                                   │
│ - errors_total                                          │
│ - bytes_transferred                                     │
├─────────────────────────────────────────────────────────┤
│ Gauge: Point-in-time value, can go up/down             │
│ - current_connections                                   │
│ - queue_depth                                           │
│ - temperature                                           │
├─────────────────────────────────────────────────────────┤
│ Histogram: Distribution of values                       │
│ - request_duration_seconds                              │
│ - response_size_bytes                                   │
│ Provides: count, sum, buckets                           │
├─────────────────────────────────────────────────────────┤
│ Summary: Similar to histogram, calculates quantiles     │
│ - request_latency_seconds (p50, p90, p99)              │
└─────────────────────────────────────────────────────────┘

Best for:
- Trends and patterns
- Alerting on thresholds
- Dashboards
- Capacity planning

Challenges:
- No event details
- Cardinality limits
- Not request-level
```

### Traces

```text
Purpose: Request flow across services

Structure:
Trace (end-to-end request)
├── Span (API Gateway) - 200ms
│   ├── Span (Auth) - 20ms
│   └── Span (OrderService) - 150ms
│       ├── Span (Database) - 50ms
│       └── Span (PaymentService) - 80ms
│           └── Span (External API) - 60ms

Best for:
- Understanding request flow
- Finding bottlenecks
- Debugging distributed issues
- Service dependencies

Challenges:
- Storage intensive
- Requires sampling
- Complex to implement
```

## Signal Correlation

### Why Correlate?

```text
Without correlation:
- Metrics: "Error rate is high"
- Logs: "Error logs from somewhere"
- Traces: "Some traces show errors"
→ Hard to connect the dots

With correlation:
- Metrics: "Error rate spike at 10:30"
  └── Click to see: Exemplar trace
      └── Click to see: Related logs
→ Full picture in seconds
```

### Correlation Methods

```text
1. Trace ID injection:
   All signals include trace_id

   Log: {"trace_id": "abc123", "message": "..."}
   Metric: http_requests{trace_id="abc123"}
   Trace: TraceID = abc123

2. Exemplars:
   Metrics point to sample traces

   request_latency = 2.5s
   └── exemplar: trace_id=abc123
   → "Show me a slow request"

3. Time correlation:
   Align signals by timestamp

   Metric spike at 10:30
   → Query logs around 10:30
   → Query traces around 10:30
```

### Unified Query Example

```text
Investigation flow:

1. Dashboard shows latency spike
   http_request_duration_p99 = 3s

2. Click on spike → exemplar trace
   trace_id: abc123

3. View trace → slow database span
   db.query: SELECT * FROM orders... (2.5s)

4. Query logs with trace_id
   {"trace_id":"abc123","query":"SELECT...","rows":50000}

5. Root cause identified
   Missing index causing full table scan
```

## OpenTelemetry Unified Approach

```text
OpenTelemetry provides unified API for all signals:

Application Code
      │
      ▼
┌─────────────────────────────────────────────────────┐
│              OpenTelemetry SDK                       │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐             │
│  │ Tracer  │  │  Meter  │  │ Logger  │             │
│  │Provider │  │Provider │  │Provider │             │
│  └────┬────┘  └────┬────┘  └────┬────┘             │
│       │            │            │                   │
│       └────────────┼────────────┘                   │
│                    │                                │
│            ┌───────┴───────┐                        │
│            │  Exporters    │                        │
│            └───────────────┘                        │
└─────────────────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┐
     ▼               ▼               ▼
┌─────────┐   ┌─────────┐    ┌─────────┐
│  Tempo  │   │Prometheus│   │  Loki   │
│(Traces) │   │(Metrics) │   │ (Logs)  │
└─────────┘   └─────────┘    └─────────┘
```

## Logging Patterns

### Structured Logging

```text
Unstructured (bad):
"User 12345 failed to login: invalid password"

Structured (good):
{
  "event": "login_failed",
  "user_id": "12345",
  "reason": "invalid_password",
  "timestamp": "2024-01-15T10:30:00Z",
  "trace_id": "abc123"
}

Benefits:
- Queryable: user_id:12345 AND event:login_failed
- Parseable: Automated analysis
- Correlatable: trace_id links to traces
```

### Log Levels

```text
Level     | When to use
----------|------------------------------------------
TRACE     | Very detailed, development only
DEBUG     | Development, verbose
INFO      | Normal operations, audit events
WARN      | Degraded, recoverable issues
ERROR     | Failures requiring attention
FATAL     | Application cannot continue

Production typically: INFO and above
Debug mode: DEBUG and above
```

### Log Aggregation Architecture

```text
┌─────────────────────────────────────────────────────────┐
│  Application Pods                                        │
│  ┌──────┐ ┌──────┐ ┌──────┐                            │
│  │ App  │ │ App  │ │ App  │ → stdout/stderr             │
│  └──────┘ └──────┘ └──────┘                            │
└─────────────────────────────────────────────────────────┘