monitoring-ops

What this skill does

# Monitoring Operations

Comprehensive observability patterns covering the three pillars (metrics, logging, tracing), alerting strategies, dashboard design, and infrastructure monitoring for production systems.

---

## Three Pillars Quick Reference

Use this table to decide which observability signal fits your need:

| Pillar | Best For | Tools | Data Type |
|--------|----------|-------|-----------|
| **Metrics** | Aggregated numeric measurements, trends, alerting on thresholds | Prometheus, Datadog, CloudWatch, StatsD | Time-series (numeric) |
| **Logs** | Discrete events, error details, audit trails, debugging context | Loki, ELK, CloudWatch Logs, Fluentd | Unstructured/structured text |
| **Traces** | Request flow across services, latency breakdown, dependency mapping | Jaeger, Tempo, Zipkin, Datadog APM | Span trees (structured) |

**When to use which:**

- **"How many requests per second?"** → Metrics (counter + rate)
- **"Why did this specific request fail?"** → Logs (error message + stack trace)
- **"Where is the latency in this request?"** → Traces (span waterfall)
- **"Is the system healthy right now?"** → Metrics (gauges + alerts)
- **"What happened at 3:42 AM?"** → Logs (timestamped event search)
- **"Which downstream service caused the timeout?"** → Traces (span analysis)

**Correlation is key:** Connect all three by embedding `trace_id` in log entries, recording exemplars in metrics, and linking trace spans to log queries.

---

## Metrics Type Decision Tree

Use this tree to select the correct metric type:

```
What are you measuring?
│
├─ A count of events that only goes up?
│  └─ COUNTER
│     Examples: http_requests_total, errors_total, bytes_sent_total
│     Use rate() or increase() to get per-second or per-interval values
│     Never use a counter's raw value — it resets on restart
│
├─ A current value that goes up AND down?
│  └─ GAUGE
│     Examples: temperature_celsius, active_connections, queue_depth
│     Use for snapshots of current state
│     Can use avg_over_time(), max_over_time() for trends
│
├─ A distribution of values (latency, size)?
│  │
│  ├─ Need aggregatable quantiles across instances?
│  │  └─ HISTOGRAM
│  │     Examples: http_request_duration_seconds, response_size_bytes
│  │     Define buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
│  │     Use histogram_quantile() for percentiles (p50, p95, p99)
│  │     Aggregatable across instances (histograms can be summed)
│  │
│  └─ Need pre-calculated quantiles on a single instance?
│     └─ SUMMARY
│        Examples: go_gc_duration_seconds
│        Pre-calculates quantiles client-side
│        NOT aggregatable across instances
│        Prefer histogram unless you have a specific reason
│
└─ None of the above?
   └─ INFO metric (labels only, value=1)
      Examples: build_info{version="1.2.3", commit="abc123"}
      Use for metadata exposed as metrics
```

**Rule of thumb:** Start with counters and histograms. Add gauges for current state. Avoid summaries unless you have a compelling reason.

---

## Alerting Decision Tree

```
What type of alert do you need?
│
├─ Known threshold with a fixed boundary?
│  └─ THRESHOLD-BASED
│     Example: CPU > 90% for 5 minutes
│     Pros: Simple, predictable, easy to understand
│     Cons: Requires manual tuning, doesn't adapt to patterns
│     Best for: Resource limits, error rate spikes, queue depth
│
├─ Normal behavior varies by time/season?
│  └─ ANOMALY-BASED
│     Example: Traffic 3 standard deviations below normal for this hour
│     Pros: Adapts to patterns, catches novel failures
│     Cons: Noisy during transitions, requires training data
│     Best for: Traffic patterns, business metrics, gradual degradation
│
└─ Defined reliability targets?
   └─ SLO-BASED (PREFERRED)
      Example: Error budget burn rate > 14.4x for 1 hour
      Pros: Aligned with user impact, reduces noise, principled
      Cons: Requires SLI/SLO definition, more complex setup
      Best for: User-facing services, platform reliability
```

### Severity Levels

| Severity | Response | Examples | Routing |
|----------|----------|----------|---------|
| **Critical (P1)** | Page on-call immediately | Service down, data loss risk, security breach | PagerDuty high-urgency, phone call |
| **Warning (P2)** | Investigate within hours | Elevated error rate, disk 80% full, SLO burn rate elevated | PagerDuty low-urgency, Slack alert channel |
| **Info (P3)** | Review next business day | Deployment completed, certificate expiring in 30 days | Slack info channel, ticket auto-created |

### When to Page vs When to Ticket

**Page (wake someone up) when:**
- Users are currently impacted
- Data loss is occurring or imminent
- Security incident is active
- Error budget will be exhausted within hours

**Create ticket (don't page) when:**
- Issue is not user-facing yet
- Automated remediation is possible
- Degradation is slow and has runway
- Issue is during business hours and can be triaged normally

---

## Structured Logging Quick Reference

### Standard JSON Log Format

```json
{
  "timestamp": "2026-03-09T14:32:01.123Z",
  "level": "ERROR",
  "message": "Failed to process payment",
  "service": "payment-api",
  "version": "1.4.2",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": "req-abc123",
  "user_id": "usr-789",
  "error": {
    "type": "PaymentGatewayTimeout",
    "message": "Gateway response timeout after 30s",
    "stack": "..."
  },
  "duration_ms": 30042,
  "http": {
    "method": "POST",
    "path": "/api/v1/payments",
    "status_code": 504
  }
}
```

### Log Level Decision Guide

| Level | When to Use | Examples |
|-------|-------------|---------|
| **DEBUG** | Development only, verbose internal state | Variable values, SQL queries, cache hits/misses |
| **INFO** | Normal operations worth recording | Request completed, job started/finished, config loaded |
| **WARN** | Degraded but still functioning | Retry succeeded, fallback used, approaching limit |
| **ERROR** | Operation failed, needs attention | Payment failed, API call error, constraint violation |
| **FATAL** | Process cannot continue, must exit | Database unreachable at startup, invalid config, OOM |

**Rules:**
- Never log at ERROR for expected conditions (user input validation → WARN)
- Every ERROR should be actionable — if no one will act on it, use WARN
- DEBUG should be off in production by default
- INFO should not be noisy — 1-5 log lines per request, not 50

### Correlation IDs

- Generate a `request_id` (UUID v4 or ULID) at the edge/gateway
- Propagate through all internal services via headers (`X-Request-ID`)
- Include `trace_id` and `span_id` from distributed tracing
- Log all three IDs in every log entry for cross-referencing

---

## Distributed Tracing Quick Reference

### Core Concepts

- **Trace:** End-to-end journey of a request across all services
- **Span:** A single unit of work (HTTP call, DB query, function execution)
- **Context propagation:** Passing trace/span IDs between services via headers

### W3C TraceContext Header

```
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  │                                  │                  │
              │  │                                  │                  └─ flags (01=sampled)
              │  │                                  └─ parent span ID (16 hex)
              │  └─ trace ID (32 hex)
              └─ version (00)
```

### Sampling Strategies

| Strategy | How It Works | Use When |
|----------|--------------|----------|
| **Head-based (ratio)** | Decide at trace start, propagate decision | Low traffic, need predictable volume |
| **Always-on** | Sample everything | Development, low-traffic services |
| **Parent-based** | Follow parent's sampling decision | Default for most services |
| **Tail-based** | Decide after trace completes (at Collector) | Need error/slow traces, high traffic |

**Recommendation:** Use parent