sentry-incident-runbook

What this skill does

# Sentry Incident Runbook

## Overview

Structured incident response framework built on Sentry's error monitoring platform. Covers the full lifecycle from alert detection through severity classification, root cause investigation using Sentry's breadcrumbs and stack traces, Discover queries for impact analysis, stakeholder communication, resolution via the Sentry API, and postmortem documentation with Sentry data exports.

## Prerequisites

- Sentry account with project-level access and auth token (`SENTRY_AUTH_TOKEN`)
- Organization slug (`SENTRY_ORG`) and project slug (`SENTRY_PROJECT`) configured
- `@sentry/node` (v8+) or equivalent SDK installed in the application
- Alert rules configured for critical error thresholds
- Notification channels connected (Slack integration or PagerDuty)

## Instructions

### Step 1 — Classify Severity

Assign a severity level based on error frequency and user impact. This determines response time and escalation path.

| Severity | Error Criteria | User Impact | Response Time | Escalation |
|----------|---------------|-------------|---------------|------------|
| **P0 — Critical** | Crash-free rate below 95% or unhandled exception spike >500/min | Core flow blocked for all users, data loss risk | 15 minutes | PagerDuty page to on-call engineer |
| **P1 — Major** | New issue affecting >100 unique users per hour | Key feature degraded, no workaround | 1 hour | Slack `#incidents` channel, tag team lead |
| **P2 — Minor** | New issue affecting <100 unique users per hour | Feature degraded but workaround exists | Same business day | Slack `#alerts-production` |
| **P3 — Low** | Edge case, cosmetic error, staging-only issue | Minimal or no user-facing impact | Next sprint | Add to backlog, assign owner |

Decision logic for classification:

```
Alert fires →
├── Check crash-free rate (Project Settings → Crash Free Sessions)
│   └── Below 95%? → P0
├── Check unique users affected (Issue Details → Users tab)
│   ├── >100/hr on core flow? → P1
│   └── <100/hr or workaround exists? → P2
└── Staging-only or edge case? → P3
```

### Step 2 — Triage and Investigate

Execute this checklist within the first 15 minutes of a P0/P1 alert.

**Initial triage (Sentry UI):**

1. Open the Sentry issue link from the alert notification
2. Check the error frequency graph — determine if the rate is spiking, steady, or declining
3. Read the "First Seen" and "Last Seen" timestamps to determine if this is new or a regression
4. Check the "Users" count on the issue to quantify impact
5. Verify the environment filter — confirm this is production, not staging
6. Check the "Release" tag — identify which deployment introduced the error
7. Open "Suspect Commits" to find the likely-causal changeset

**Deep investigation (stack trace and breadcrumbs):**

1. Read the full stack trace — identify the failing function and line number
2. Expand the breadcrumbs panel — trace the sequence of events leading to the error (HTTP requests, console logs, navigation, UI clicks)
3. Check the user context panel for device, browser, OS, and custom user tags
4. Review the "Tags" panel for patterns (specific release, region, browser)

**API-based investigation:**

```bash
# Fetch issue details programmatically
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "
import json, sys
issue = json.load(sys.stdin)
print(f'Title:      {issue[\"title\"]}')
print(f'First Seen: {issue[\"firstSeen\"]}')
print(f'Last Seen:  {issue[\"lastSeen\"]}')
print(f'Events:     {issue[\"count\"]}')
print(f'Users:      {issue[\"userCount\"]}')
print(f'Level:      {issue[\"level\"]}')
print(f'Status:     {issue[\"status\"]}')
print(f'Platform:   {issue.get(\"platform\", \"unknown\")}')
" || echo "ERROR: Failed to fetch issue — check SENTRY_AUTH_TOKEN and ISSUE_ID"

# Fetch latest events for the issue (most recent 5)
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/events/?per_page=5" \
  | python3 -c "
import json, sys
events = json.load(sys.stdin)
for e in events:
    release = e.get('release', {})
    ver = release.get('version', 'N/A') if isinstance(release, dict) else 'N/A'
    print(f'Event {e[\"eventID\"][:12]} | {e.get(\"dateCreated\", \"N/A\")} | Release: {ver}')
" || echo "ERROR: Failed to fetch events"
```

**Sentry Discover queries for impact analysis:**

```bash
# Count total events and unique affected users in last 24 hours
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/events/" \
  --data-urlencode "field=count()" \
  --data-urlencode "field=count_unique(user)" \
  --data-urlencode "query=issue.id:$ISSUE_ID" \
  --data-urlencode "statsPeriod=24h" \
  -G | python3 -c "
import json, sys
data = json.load(sys.stdin)
if 'data' in data and data['data']:
    row = data['data'][0]
    print(f'Events (24h):       {row.get(\"count()\", \"N/A\")}')
    print(f'Unique users (24h): {row.get(\"count_unique(user)\", \"N/A\")}')
" || echo "ERROR: Discover query failed"

# Check p95 transaction duration for affected endpoint
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/events/" \
  --data-urlencode "field=transaction" \
  --data-urlencode "field=p95(transaction.duration)" \
  --data-urlencode "field=count()" \
  --data-urlencode "query=has:transaction event.type:transaction" \
  --data-urlencode "statsPeriod=1h" \
  --data-urlencode "sort=-count()" \
  --data-urlencode "per_page=5" \
  -G | python3 -c "
import json, sys
data = json.load(sys.stdin)
if 'data' in data:
    for row in data['data']:
        txn = row.get('transaction', 'unknown')
        p95 = row.get('p95(transaction.duration)', 0)
        cnt = row.get('count()', 0)
        print(f'{txn}: p95={p95:.0f}ms, count={cnt}')
" || echo "ERROR: Transaction query failed"
```

### Step 3 — Resolve, Communicate, and Document

**Identify the root cause pattern:**

| Pattern | Diagnostic Signal | Immediate Action |
|---------|------------------|-----------------|
| Deployment regression | "First Seen" aligns with latest deploy timestamp | Rollback via `sentry-cli releases deploys $PREV_VERSION new --env production` |
| Third-party failure | Breadcrumbs show failed HTTP calls to external hosts | Enable circuit breaker, add retry logic, monitor dependency status |
| Data corruption | Event context contains malformed input samples | Add input validation, fix data pipeline upstream |
| Resource exhaustion | Error rate correlates with traffic spikes (OOM, pool exhaustion) | Scale horizontally, add connection pooling, implement rate limiting |
| SDK misconfiguration | Events missing context, breadcrumbs, or release info | Review `Sentry.init()` options, verify source maps uploaded |

**Resolve the issue via Sentry API:**

```bash
# Mark issue as resolved (closes the issue)
curl -s -X PUT \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved"}' \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Status: {d.get(\"status\",\"unknown\")}')" \
  || echo "ERROR: Failed to resolve issue"

# Resolve in next release (auto-reopens on regression)
curl -s -X PUT \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved", "statusDetails": {"inNextRelease": true}}' \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Status: {d.get(\"status\",\"unknown\")} (regression detection enabled)')" \
  || echo "ERROR: Failed to resolve issue"

# Ignore with threshold (snooze until count exceeds limit)
# Use 100 as the re-alert threshold for noisy low-severity issues
curl -s -X PUT \
  -H "