instrumentation-planning

What this skill does

# Instrumentation Planning

Strategic planning for application instrumentation before implementation.

## When to Use This Skill

- Planning instrumentation for new services
- Reviewing instrumentation strategy
- Establishing naming conventions
- Managing telemetry cardinality
- Setting instrumentation budgets

## Instrumentation Strategy Framework

### What to Instrument

```text
Instrumentation Layers:

┌─────────────────────────────────────────────────────────────────┐
│  Layer 1: Automatic/Library Instrumentation                     │
│  - HTTP clients/servers (auto-captured)                         │
│  - Database clients (auto-captured)                             │
│  - Message queue clients (auto-captured)                        │
│  - Framework-provided metrics                                   │
│  Effort: Low | Coverage: Broad | Customization: Limited        │
├─────────────────────────────────────────────────────────────────┤
│  Layer 2: Business Transaction Instrumentation                  │
│  - Key user journeys                                            │
│  - Business operations (checkout, signup, etc.)                │
│  - Revenue-generating flows                                     │
│  - SLA-bound operations                                         │
│  Effort: Medium | Coverage: Targeted | Value: High             │
├─────────────────────────────────────────────────────────────────┤
│  Layer 3: Debug/Diagnostic Instrumentation                      │
│  - Algorithmic hot paths                                        │
│  - Cache behavior                                               │
│  - Circuit breaker states                                       │
│  - Retry/fallback paths                                         │
│  Effort: Medium | Coverage: Deep | Use: Troubleshooting        │
├─────────────────────────────────────────────────────────────────┤
│  Layer 4: Business Metrics                                      │
│  - Domain-specific counters                                     │
│  - Conversion rates                                             │
│  - Feature usage                                                │
│  - Customer behavior                                            │
│  Effort: High | Coverage: Custom | Value: Business Insights    │
└─────────────────────────────────────────────────────────────────┘
```

### Instrumentation Decision Matrix

```yaml
instrumentation_decisions:
  always_instrument:
    - "Inbound HTTP/gRPC requests"
    - "Outbound HTTP/gRPC calls"
    - "Database queries"
    - "Message publish/consume"
    - "Authentication/authorization"
    - "External API calls"
    - "Cache operations"

  consider_instrumenting:
    - "Complex business logic"
    - "Feature flags evaluation"
    - "Background jobs"
    - "Scheduled tasks"
    - "File I/O operations"
    - "CPU-intensive operations"

  avoid_instrumenting:
    - "Every method call (too noisy)"
    - "Tight loops (performance impact)"
    - "Data transformation (low value)"
    - "Validation helpers"
    - "Utility functions"

  decision_criteria:
    business_value:
      weight: 0.3
      question: "Does this help understand business outcomes?"

    debugging_value:
      weight: 0.25
      question: "Does this help diagnose production issues?"

    slo_relevance:
      weight: 0.25
      question: "Does this contribute to SLI measurement?"

    cost_impact:
      weight: 0.2
      question: "Is the cardinality/volume acceptable?"
```

## Naming Conventions

### Metric Naming

```yaml
metric_naming:
  format: "[namespace]_[subsystem]_[name]_[unit]"

  rules:
    case: "snake_case"
    unit_suffix: "Always include (_seconds, _bytes, _total)"
    base_units: "Use base units (seconds not milliseconds)"
    counter_suffix: "_total for counters"

  examples:
    good:
      - "http_server_requests_total"
      - "http_server_request_duration_seconds"
      - "http_server_response_size_bytes"
      - "db_connections_current"
      - "order_processing_duration_seconds"
      - "payment_transactions_total"

    bad:
      - "requests (no unit, no namespace)"
      - "HttpRequestDuration (wrong case)"
      - "order_latency_ms (use base units)"
      - "totalOrders (camelCase, no unit)"

  label_naming:
    case: "snake_case"
    avoid:
      - "Embedded values in name (path=/users)"
      - "High cardinality labels"
    good_labels:
      - "method, status_code, path"
      - "service, version, environment"
    bad_labels:
      - "user_id (high cardinality)"
      - "request_id (high cardinality)"
      - "timestamp (not a dimension)"
```

### Span Naming

```yaml
span_naming:
  format: "[operation] [resource]"

  rules:
    - "Use verb + noun pattern"
    - "Keep names low cardinality"
    - "Include operation type, not specific values"
    - "Be consistent across services"

  examples:
    http:
      pattern: "HTTP {METHOD} {route_template}"
      good: "HTTP GET /users/{id}"
      bad: "HTTP GET /users/12345"

    database:
      pattern: "{operation} {table}"
      good: "SELECT orders"
      bad: "SELECT * FROM orders WHERE id=123"

    messaging:
      pattern: "{operation} {queue/topic}"
      good: "PUBLISH order-events"
      bad: "publish message to order-events queue"

    rpc:
      pattern: "{service}/{method}"
      good: "OrderService/CreateOrder"
      bad: "grpc call to order service"

  attributes:
    required:
      - "service.name"
      - "service.version"
      - "deployment.environment"

    recommended:
      http:
        - "http.method"
        - "http.route"
        - "http.status_code"
        - "http.target"

      database:
        - "db.system"
        - "db.name"
        - "db.operation"
        - "db.statement (sanitized)"

      messaging:
        - "messaging.system"
        - "messaging.destination"
        - "messaging.operation"
```

### Log Field Naming

```yaml
log_naming:
  format: "snake_case for all fields"

  standard_fields:
    timestamp: "ISO 8601 format"
    level: "INFO, WARN, ERROR, etc."
    message: "Human-readable description"
    service: "Service name"
    trace_id: "Correlation ID"
    span_id: "Current span"

  domain_fields:
    pattern: "{domain}_{field}"
    examples:
      - "order_id"
      - "customer_id"
      - "payment_amount"
      - "product_sku"

  avoid:
    - "Nested objects (flatten for indexing)"
    - "Arrays of unknown length"
    - "Large text blobs"
    - "Sensitive data (PII, secrets)"
```

## Cardinality Management

### Understanding Cardinality

```text
Cardinality = Number of unique time series

Example:
http_requests_total{method="GET", path="/api/users", status="200"}

Cardinality = methods × paths × statuses
            = 5 × 100 × 10
            = 5,000 time series

With user_id (1M users):
            = 5 × 100 × 10 × 1,000,000
            = 5,000,000,000 time series ← EXPLOSION!
```

### Cardinality Budget

```yaml
cardinality_budget:
  planning:
    total_budget: 100000  # Target max time series per service
    allocation:
      automatic_instrumentation: 30%  # 30,000
      business_transactions: 40%      # 40,000
      custom_metrics: 20%             # 20,000
      buffer: 10%                     # 10,000

  per_metric_limits:
    low_cardinality:
      max_series: 100
      example: "status codes, methods"

    medium_cardinality:
      max_series: 1000
      example: "endpoints, operations"

    high_cardinality:
      max_series: 10000
      example: "aggregated by hour"
      requires: "Justification and approval"

  monitoring:
    - "Alert on cardinality growth > 10% per day"
    - "Weekly cardinality reviews"
    - "Automatic label value limiting"
```

### Cardinality Reduction Techniques

```yaml
cardinality_reduction:
  bucketing:
    before: "path=/users/12345"
    after: "path=/users/{id}"
    technique: "Path template extraction"

  sampling:
    description: "Sample high-volume, low-value traces"
    strategies:
      head_sampling: "Decide at tra