observability-monitoring

What this skill does

# Observability & Monitoring

A comprehensive skill for implementing production-grade observability and monitoring using Prometheus, Grafana, and the wider cloud-native monitoring ecosystem. This skill covers metrics collection, time-series analysis, alerting, visualization, and operational excellence patterns.

## When to Use This Skill

Use this skill when:

- Setting up monitoring for production systems and applications
- Implementing metrics collection and observability for microservices
- Creating dashboards and visualizations for system health monitoring
- Defining alerting rules and incident response automation
- Analyzing system performance and capacity using time-series data
- Implementing SLIs, SLOs, and SLAs for service reliability
- Debugging production issues using metrics and traces
- Building custom exporters for application-specific metrics
- Setting up federation for multi-cluster monitoring
- Migrating from legacy monitoring to cloud-native solutions
- Implementing cost monitoring and optimization tracking
- Creating real-time operational dashboards for DevOps teams

## Core Concepts

### The Four Pillars of Observability

Modern observability is built on four fundamental pillars:

1. **Metrics**: Numerical measurements of system behavior over time
   - Counter: Monotonically increasing values (requests served, errors)
   - Gauge: Point-in-time values that go up and down (memory usage, temperature)
   - Histogram: Distribution of values (request duration buckets)
   - Summary: Similar to histogram but calculates quantiles on client-side

2. **Logs**: Discrete events with contextual information
   - Structured logging (JSON, key-value pairs)
   - Centralized log aggregation (ELK, Loki)
   - Correlation with metrics and traces

3. **Traces**: Request flow through distributed systems
   - Span: Single unit of work with start/end time
   - Trace: Collection of spans representing end-to-end request
   - OpenTelemetry for distributed tracing

4. **Events**: Significant occurrences in system lifecycle
   - Deployments, configuration changes
   - Scaling events, incidents
   - Business events and user actions

### Prometheus Architecture

Prometheus is a pull-based monitoring system with key components:

**Time-Series Database (TSDB)**
- Stores metrics as time-series data
- Efficient compression and retention policies
- Local storage with optional remote storage

**Scrape Targets**
- Service discovery (Kubernetes, Consul, EC2, etc.)
- Static configuration
- Relabeling for flexible target selection

**PromQL Query Engine**
- Powerful query language for metrics analysis
- Aggregation, filtering, and mathematical operations
- Range vectors and instant vectors

**Alertmanager**
- Alert rule evaluation
- Grouping, silencing, and routing
- Integration with PagerDuty, Slack, email, webhooks

**Exporters**
- Bridge between Prometheus and systems
- Node exporter, cAdvisor, custom exporters
- Third-party exporters for databases, services

### Metric Labels and Cardinality

Labels are key-value pairs attached to metrics:

```prometheus
http_requests_total{method="GET", endpoint="/api/users", status="200"}
```

**Label Best Practices:**
- Use labels for dimensions you query/aggregate by
- Avoid high-cardinality labels (user IDs, timestamps)
- Keep label names consistent across metrics
- Use relabeling to normalize external labels

**Cardinality Considerations:**
- Each unique label combination = new time-series
- High cardinality = increased memory and storage
- Monitor cardinality with `prometheus_tsdb_symbol_table_size_bytes`
- Use recording rules to pre-aggregate high-cardinality metrics

### Recording Rules

Pre-compute frequently-used or expensive queries:

```yaml
groups:
  - name: api_performance
    interval: 30s
    rules:
      - record: api:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
      - record: api:http_requests:rate5m
        expr: rate(http_requests_total[5m])
```

**Benefits:**
- Faster dashboard loading
- Reduced query load on Prometheus
- Consistent metric naming conventions
- Enable complex aggregations

### Service Level Objectives (SLOs)

Define and track reliability targets:

**SLI (Service Level Indicator)**: Metric measuring service quality
- Availability: % of successful requests
- Latency: % of requests under threshold
- Throughput: Requests per second

**SLO (Service Level Objective)**: Target for SLI
- 99.9% availability (43.8 minutes downtime/month)
- 95% of requests < 200ms
- 1000 RPS sustained

**SLA (Service Level Agreement)**: Contract with consequences
- External commitments to customers
- Financial penalties for SLO violations

**Error Budget**: Acceptable failure rate
- Error budget = 100% - SLO
- 99.9% SLO = 0.1% error budget
- Use budget for innovation vs. reliability tradeoff

## Prometheus Setup and Configuration

### Basic Prometheus Configuration

```yaml
# prometheus.yml
global:
  scrape_interval: 15s      # Default scrape interval
  evaluation_interval: 15s  # Alert rule evaluation interval
  external_labels:
    cluster: 'production'
    region: 'us-west-2'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - 'rules/*.yml'
  - 'alerts/*.yml'

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter for system metrics
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
          - 'node3:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '${1}'

  # Application metrics
  - job_name: 'api'
    static_configs:
      - targets: ['api-1:8080', 'api-2:8080', 'api-3:8080']
        labels:
          env: 'production'
          tier: 'backend'
```

### Kubernetes Service Discovery

```yaml
scrape_configs:
  # Kubernetes API server
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

  # Kubernetes pods with prometheus.io annotations
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with prometheus.io/scrape: "true" annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the port from prometheus.io/port annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (\d+)
        target_label: __address__
        replacement: ${1}:${2}
      # Use the path from prometheus.io/path annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
      # Add namespace label
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      # Add pod name label
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Kubernetes services
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_la