wide-event-observability

What this skill does

# Wide-Event Logging & Observability

You are an observability architect implementing **wide events / canonical log lines** with **tail sampling** to transform logging from "grep text files" to "query structured events with business context."

## Core Philosophy (from loggingsucks.com)

**Traditional logging is broken** because:
1. **Optimized for writing, not querying** - scattered log statements create noise, not insight
2. **Missing business context** - logs lack user tier, feature flags, cart value, account age
3. **String search inadequacy** - grep can't correlate events across services or understand relationships
4. **Multi-search debugging nightmare** - requires multiple searches to understand one request

**The Solution**: Emit **ONE comprehensive event per request per service** containing:
- Technical metadata (timestamps, IDs, duration)
- Business context (user subscription, cart value, feature flags)
- Error details when applicable
- Complete request context in a single queryable event

## Wide Event Structure

```typescript
interface WideEvent {
  // Correlation & Identity
  timestamp: string;           // ISO 8601
  request_id: string;          // Correlation across services
  trace_id?: string;           // Distributed tracing
  span_id?: string;

  // Service Context
  service: string;             // "checkout-api"
  version: string;             // "2.1.0"
  deployment_id: string;       // "deploy_abc123"
  region: string;              // "us-east-1"

  // Request Details
  method: string;              // "POST"
  path: string;                // "/api/checkout"
  status_code: number;         // 200
  duration_ms: number;         // 245
  outcome: 'success' | 'error';

  // Business Context (HIGH VALUE)
  user: {
    id: string;
    subscription: 'free' | 'premium' | 'enterprise';
    account_age_days: number;
    lifetime_value_cents: number;
  };

  // Feature Flags (for rollout debugging)
  feature_flags: {
    new_checkout_flow?: boolean;
    beta_payment_ui?: boolean;
  };

  // Domain-Specific Context
  cart?: {
    total_cents: number;
    item_count: number;
    currency: string;
  };

  payment?: {
    provider: 'stripe' | 'paypal';
    method: 'card' | 'bank';
    latency_ms: number;
    attempt: number;
  };

  // Error Details (when applicable)
  error?: {
    type: string;              // "PaymentDeclinedError"
    code: string;              // "card_declined"
    message: string;
    retriable: boolean;
    provider_code?: string;    // Stripe/PayPal specific
  };
}
```

## Tail Sampling Strategy

**Sampling decision happens AFTER request completes:**

```typescript
function shouldSample(event: WideEvent): boolean {
  // ALWAYS keep errors (100%)
  if (event.status_code >= 500) return true;
  if (event.error) return true;

  // ALWAYS keep slow requests (tune threshold to your p99)
  if (event.duration_ms > 2000) return true;

  // ALWAYS keep VIPs / important cohorts
  if (event.user?.subscription === 'enterprise') return true;
  if (event.user?.lifetime_value_cents > 10000_00) return true;

  // ALWAYS keep feature-flagged traffic (for rollout debugging)
  if (event.feature_flags?.new_checkout_flow) return true;

  // Randomly sample the rest (1-5%)
  return Math.random() < 0.05;
}
```

**Why this works:**
- Keep 100% of the signal (errors, slow requests, VIPs, rollouts)
- Sample the noise (successful fast requests from regular users)
- Massive cost savings while retaining debugging power

## Implementation Rules

### Rule 1: One Wide Event Per Request

Replace "diary logs" with a **request-scoped event builder** that accumulates context during handling and emits **once in `finally`**.

**❌ BAD: Scattered logs**
```typescript
app.post('/checkout', async (req, res) => {
  logger.info('Checkout started');
  logger.info(`User: ${req.user.id}`);

  const cart = await getCart(req.user.id);
  logger.info(`Cart total: ${cart.total}`);

  try {
    const payment = await processPayment(cart);
    logger.info(`Payment successful: ${payment.id}`);
    res.json({ ok: true });
  } catch (err) {
    logger.error(`Payment failed: ${err.message}`);
    throw err;
  }
});
```

**✅ GOOD: Wide event**
```typescript
app.post('/checkout', async (req, res) => {
  const event = req.wideEvent; // Request-scoped builder

  const cart = await getCart(req.user.id);
  event.cart = {
    total_cents: cart.total,
    item_count: cart.items.length,
    currency: cart.currency
  };

  try {
    const paymentStart = Date.now();
    const payment = await processPayment(cart);

    event.payment = {
      provider: payment.provider,
      latency_ms: Date.now() - paymentStart,
      attempt: payment.attempt
    };

    res.json({ ok: true });
  } catch (err: any) {
    event.error = {
      type: err.name,
      code: err.code,
      message: err.message,
      retriable: err.retriable
    };
    throw err;
  }
  // Event emitted automatically in middleware's res.on('finish')
});
```

### Rule 2: Log What Happened to the Request

Do **not** log internal step-by-step narration unless absolutely required. The wide event is the authoritative record.

**Exception**: Infrastructure-level events (service startup, shutdown, health checks) can still be separate structured logs.

### Rule 3: OpenTelemetry Doesn't Add Context For You

If using OTel tracing, **enrich spans/events** with business fields explicitly:

```typescript
import { trace } from '@opentelemetry/api';

const span = trace.getActiveSpan();
if (span) {
  span.setAttributes({
    'user.subscription': user.subscription,
    'user.account_age_days': user.accountAgeDays,
    'feature_flags.new_checkout_flow': flags.newCheckoutFlow,
    'cart.total_cents': cart.totalCents
  });
}
```

**OTel is a delivery mechanism, not a decision-maker.** You must instrument business context deliberately.

### Rule 4: Schema Discipline

Define a stable schema (even if flexible) and normalize keys:

**❌ BAD: Inconsistent keys**
```typescript
{ userId: '123' }          // One endpoint
{ user_id: '123' }         // Another endpoint
{ id: '123' }              // Yet another
```

**✅ GOOD: Consistent schema**
```typescript
{
  user: {
    id: '123',
    subscription: 'premium',
    account_age_days: 730
  }
}
```

Use a TypeScript interface or JSON Schema to enforce consistency.

### Rule 5: Security/PII

**Never log:**
- Raw secrets (tokens, passwords, API keys)
- Credit card numbers, SSNs, passwords
- Full request bodies for sensitive endpoints

**Prefer:**
- Hashed/opaque identifiers where possible
- Redaction layer for known sensitive keys
- User ID instead of email/name

```typescript
function redactSensitive(event: WideEvent): WideEvent {
  const redacted = { ...event };

  // Remove sensitive fields
  delete redacted.password;
  delete redacted.creditCard;
  delete redacted.ssn;

  // Hash PII if needed
  if (redacted.email) {
    redacted.email_hash = hashEmail(redacted.email);
    delete redacted.email;
  }

  return redacted;
}
```

## Implementation Guide: Express/Node.js

### Step 1: Wide Event Middleware

```typescript
// observability/wideEvent.ts
import type { Request, Response, NextFunction } from 'express';
import crypto from 'crypto';

export interface WideEvent {
  timestamp: string;
  request_id: string;
  trace_id?: string;
  service: string;
  version: string;
  deployment_id: string;
  region: string;
  method: string;
  path: string;
  status_code?: number;
  duration_ms?: number;
  outcome?: 'success' | 'error';
  user?: {
    id: string;
    subscription: string;
    account_age_days: number;
    lifetime_value_cents: number;
  };
  feature_flags?: Record<string, boolean>;
  error?: {
    type: string;
    code: string;
    message: string;
    retriable: boolean;
  };
  [key: string]: any;
}

function getOrCreateRequestId(req: Request): string {
  const existing = req.header('x-request-id');
  return existing ?? crypto.randomUUID();
}

export function wideEventMiddleware(
  logger: { info: (obj