monitoring-setup

What this skill does

# Monitoring Setup

## When to Use

Activate this skill when:
- Setting up structured logging for a Python/FastAPI application
- Configuring Prometheus metrics collection and custom counters/histograms
- Implementing health check endpoints (liveness and readiness)
- Designing alert rules and thresholds for production services
- Creating Grafana dashboards for service monitoring
- Integrating Sentry for error tracking and performance monitoring
- Implementing distributed tracing with OpenTelemetry
- Reviewing or improving existing observability coverage

**Output:** Write observability configuration summary to `monitoring-config.md` documenting what was set up (metrics, alerts, dashboards, health checks).

Do NOT use this skill for:
- Responding to active production incidents (use `incident-response`)
- Deploying monitoring infrastructure (use `deployment-pipeline`)
- Writing application business logic (use `python-backend-expert`)
- Docker container configuration (use `docker-best-practices`)

## Instructions

### Four Pillars of Observability

Every production service must implement all four pillars.

```
┌─────────────────────────────────────────────────────────────────┐
│                    OBSERVABILITY                                │
├────────────────┬───────────────┬──────────────┬────────────────┤
│    METRICS     │   LOGGING     │   TRACING    │   ALERTING     │
│                │               │              │                │
│  Prometheus    │  structlog    │ OpenTelemetry│  Alert rules   │
│  counters,     │  structured   │ distributed  │  thresholds,   │
│  histograms,   │  JSON logs,   │ trace spans, │  notification  │
│  gauges        │  context      │ correlation  │  channels      │
├────────────────┴───────────────┴──────────────┴────────────────┤
│                    DASHBOARDS (Grafana)                         │
│        Visualize metrics, logs, and traces in one place        │
└─────────────────────────────────────────────────────────────────┘
```

### Pillar 1: Metrics (Prometheus)

Use the RED method for request-driven services and USE method for resources.

**RED Method (for every API endpoint):**
- **R**ate -- Requests per second
- **E**rrors -- Failed requests per second
- **D**uration -- Request latency distribution

**USE Method (for infrastructure resources):**
- **U**tilization -- Percentage of resource used (CPU, memory, disk)
- **S**aturation -- Work queued or waiting (connection pool, queue depth)
- **E**rrors -- Error events (OOM kills, connection failures)

**Key metrics to instrument:**

```python
from prometheus_client import Counter, Histogram, Gauge, Info

# RED metrics
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    labelnames=["method", "endpoint", "status_code"],
)

REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration in seconds",
    labelnames=["method", "endpoint"],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)

# USE metrics
DB_POOL_USAGE = Gauge(
    "db_connection_pool_usage",
    "Database connection pool utilization",
    labelnames=["pool_name"],
)

DB_POOL_SIZE = Gauge(
    "db_connection_pool_size",
    "Database connection pool max size",
    labelnames=["pool_name"],
)

REDIS_CONNECTIONS = Gauge(
    "redis_active_connections",
    "Active Redis connections",
)

# Business metrics
ACTIVE_USERS = Gauge(
    "active_users_total",
    "Currently active users",
)

APP_INFO = Info(
    "app",
    "Application metadata",
)
```

**FastAPI middleware for automatic metrics:**

```python
import time
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request

class PrometheusMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        method = request.method
        endpoint = request.url.path
        start_time = time.perf_counter()

        response = await call_next(request)

        duration = time.perf_counter() - start_time
        status_code = str(response.status_code)

        REQUEST_COUNT.labels(
            method=method, endpoint=endpoint, status_code=status_code
        ).inc()

        REQUEST_DURATION.labels(
            method=method, endpoint=endpoint
        ).observe(duration)

        return response
```

See `references/metrics-config-template.py` for the complete setup.

### Pillar 2: Logging (structlog)

Use structured JSON logging with contextual information. Never use `print()` or unstructured logging in production.

**Logging principles:**
1. **Structured** -- JSON format, machine-parseable
2. **Contextual** -- Include request ID, user ID, trace ID in every log
3. **Leveled** -- Use appropriate log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)
4. **Actionable** -- Every WARNING/ERROR log should indicate what to investigate

**Log levels and when to use them:**

| Level | When to Use | Example |
|-------|-------------|---------|
| DEBUG | Detailed diagnostic info, disabled in production | `Processing item 42 of 100` |
| INFO | Normal operations, significant events | `User created`, `Payment processed` |
| WARNING | Unexpected but handled situation | `Retry attempt 2 of 3`, `Cache miss` |
| ERROR | Operation failed, needs attention | `Database query failed`, `External API timeout` |
| CRITICAL | System-level failure, immediate action | `Cannot connect to database`, `Out of memory` |

**structlog setup:**

```python
import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        structlog.processors.JSONRenderer(),
    ],
    wrapper_class=structlog.stdlib.BoundLogger,
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(),
    cache_logger_on_first_use=True,
)
```

**Adding request context:**

```python
from starlette.middleware.base import BaseHTTPMiddleware
import structlog
import uuid

class LoggingContextMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            request_id=request_id,
            method=request.method,
            path=request.url.path,
        )
        response = await call_next(request)
        response.headers["X-Request-ID"] = request_id
        return response
```

See `references/logging-config-template.py` for the complete setup.

### Pillar 3: Tracing (OpenTelemetry)

Distributed tracing connects logs and metrics across service boundaries.

**Trace setup for FastAPI:**

```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def setup_tracing(app, service_name: str = "backend"):
    resource = Resource.create({"service.name": service_name})
    provider = TracerProvider(resource=resource)

    exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
    provider.add_span_processor(BatchSpanProcessor(exporter))

    trace.set_tracer_provider(provider)

    # Auto-instrument FastAPI, SQLAlchemy, Redis
    FastAPIInstrumentor.instrument_app(app)
    SQLAlchemyInstrumentor().instrument()