runbook-generator

What this skill does

# Runbook Generator

**Tier:** POWERFUL
**Category:** Engineering / SRE
**Maintainer:** Claude Skills Team

## Overview

Analyze a codebase and generate production-grade operational runbooks with copy-paste commands, verification checks after every step, rollback procedures for every destructive action, escalation paths with contact information, and time estimates for capacity planning. Detects the stack (CI/CD, database, hosting, containers) and produces runbooks tailored to the actual infrastructure. Includes staleness detection to flag runbooks when referenced config files change.

## Keywords

runbook, operational procedures, incident response, deployment, rollback, database maintenance, scaling, monitoring, on-call, SRE, postmortem

## Core Capabilities

### 1. Stack Detection
- Identify CI/CD platform, database, hosting, and orchestration from repo files
- Map detected stack to appropriate runbook templates
- Extract connection strings, deployment commands, and infrastructure details

### 2. Runbook Types
- Deployment: pre-checks, deploy steps, smoke tests, rollback
- Incident response: triage, diagnose, mitigate, resolve, postmortem
- Database maintenance: backup, migration, vacuum, reindex
- Scaling: horizontal and vertical scaling procedures
- Monitoring: alert setup, dashboard configuration, on-call rotation

### 3. Format Discipline
- Numbered steps with copy-paste commands
- Verification check after EVERY step
- Time estimates for capacity planning
- Rollback procedure for every destructive action
- Escalation paths with decision criteria

### 4. Maintenance
- Staleness detection linked to config file modification dates
- Quarterly review cadence
- Staging dry-run validation framework

## When to Use

- Codebase has no runbooks and you need to bootstrap them
- Existing runbooks are outdated or incomplete
- Onboarding a new engineer for on-call rotation
- Preparing for an incident response drill
- Post-incident improvement: updating runbooks with lessons learned

## Stack Detection

Scan the repository before writing any runbook:

```bash
# CI/CD Platform
[ -d ".github/workflows" ] && echo "GitHub Actions"
[ -f ".gitlab-ci.yml" ]    && echo "GitLab CI"
[ -f "Jenkinsfile" ]       && echo "Jenkins"

# Database
grep -rl "postgres\|postgresql" package.json pyproject.toml 2>/dev/null && echo "PostgreSQL"
grep -rl "mysql\|mariadb" package.json 2>/dev/null && echo "MySQL"
grep -rl "mongodb\|mongoose" package.json 2>/dev/null && echo "MongoDB"

# Hosting
[ -f "vercel.json" ]       && echo "Vercel"
[ -f "fly.toml" ]          && echo "Fly.io"
[ -f "railway.toml" ]      && echo "Railway"
[ -d "terraform" ]         && echo "Terraform (custom cloud)"
[ -d "k8s" ] || [ -d "kubernetes" ] && echo "Kubernetes"
[ -f "docker-compose.yml" ] && echo "Docker Compose"

# Framework
[ -f "next.config.mjs" ] || [ -f "next.config.ts" ] && echo "Next.js"
grep -q "fastapi" requirements.txt 2>/dev/null && echo "FastAPI"
[ -f "go.mod" ] && echo "Go"
```

## Deployment Runbook Template

```markdown
# Deployment Runbook — [App Name]

**Stack:** [Framework] + [Database] + [Hosting]
**Last verified:** YYYY-MM-DD
**Owner:** [Team Name]
**Estimated total time:** 15-25 minutes

---

## Staleness Check

| Config File | Last Modified | Affects Steps |
|-------------|--------------|---------------|
| vercel.json | `git log -1 --format=%ci -- vercel.json` | Deploy, Rollback |
| db/schema.ts | `git log -1 --format=%ci -- db/schema.ts` | Migration |
| .github/workflows/deploy.yml | `git log -1 --format=%ci -- .github/workflows/deploy.yml` | CI |

If any config was modified after "Last verified" date, review affected steps.

---

## Pre-Deployment Checklist
- [ ] All PRs merged to main
- [ ] CI passing on main branch
- [ ] Database migrations tested in staging
- [ ] Rollback plan confirmed
- [ ] On-call engineer notified

## Step 1: Verify CI Status (2 min)

```bash
# Check latest CI run
gh run list --branch main --limit 3

# Verify specific run
gh run view <run-id>
```

VERIFY: Latest run shows green checkmark. If red, do not proceed.

## Step 2: Apply Database Migrations (5 min)

```bash
# Staging first
DATABASE_URL=$STAGING_DB_URL pnpm db:migrate

# Verify migration applied
DATABASE_URL=$STAGING_DB_URL pnpm db:migrate status
```

VERIFY: Output shows "All migrations applied" with today's date.

```bash
# Production (only after staging verification)
DATABASE_URL=$PROD_DB_URL pnpm db:migrate
```

VERIFY: Same output as staging. If error, see Rollback section.

WARNING: For migrations on tables with >1M rows, schedule during low-traffic window and monitor lock wait times.

## Step 3: Deploy to Production (5 min)

```bash
# Option A: Git push triggers deployment
git push origin main

# Option B: Manual trigger
vercel --prod
# or: fly deploy
# or: kubectl apply -f k8s/deployment.yaml
```

VERIFY: Deployment dashboard shows new version in progress. Note the deployment URL/ID for rollback.

## Step 4: Smoke Test (5 min)

```bash
# Health check
curl -sf https://myapp.com/api/health | jq .

# Critical user path
curl -sf https://myapp.com/api/v1/me \
  -H "Authorization: Bearer $TEST_TOKEN" | jq '.id'

# Check error rate (wait 2 minutes for data)
# Dashboard: [link to monitoring dashboard]
```

VERIFY:
- Health returns `{"status": "ok", "db": "connected"}`
- User endpoint returns a valid user ID
- Error rate < 1% on monitoring dashboard

## Step 5: Monitor (10 min)

Watch these metrics for 10 minutes after deployment:
- Error rate: < 1% (dashboard: [link])
- P95 latency: < 200ms (dashboard: [link])
- Active DB connections: < 80% of max (query below)

```bash
psql $PROD_DB_URL -c "SELECT count(*) FROM pg_stat_activity;"
```

VERIFY: All metrics within normal range. If any spike, proceed to Rollback.

---

## Rollback

If smoke tests fail or metrics degrade:

```bash
# Instant rollback via Vercel
vercel rollback [previous-deployment-url]

# or Fly.io
fly releases --app myapp
fly deploy --image [previous-image]

# or Kubernetes
kubectl rollout undo deployment/myapp

# Database rollback (ONLY if migration was applied in this deploy)
DATABASE_URL=$PROD_DB_URL pnpm db:rollback
```

VERIFY: Previous version serving traffic. Run smoke tests again.

---

## Escalation

| Level | Who | When | Contact |
|-------|-----|------|---------|
| L1 | On-call engineer | First responder | PagerDuty rotation |
| L2 | Platform lead | DB issues, rollback failures | Slack: @platform-lead |
| L3 | VP Engineering | Production down > 30 min | Phone: [number] |
```

## Incident Response Runbook Template

```markdown
# Incident Response Runbook

**Severity:** P1 (down), P2 (degraded), P3 (minor)
**Estimated time:** P1: 30-60 min, P2: 1-4 hours, P3: next business day

---

## Phase 1: Triage (5 min)

### Confirm the Incident

```bash
# Is the app responding?
curl -sw "%{http_code}" https://myapp.com/api/health -o /dev/null

# Check for errors in recent logs
vercel logs --since=15m | grep -i "error\|exception\|5[0-9][0-9]"
# or: kubectl logs -l app=myapp --since=15m | grep -i error
```

VERIFY: 200 = app is up. 5xx or timeout = incident confirmed.

### Declare Severity

| Condition | Severity | Action |
|-----------|----------|--------|
| Site completely unreachable | P1 | Page L2/L3 immediately |
| Partial degradation or slow | P2 | Notify team channel |
| Single feature broken | P3 | Create ticket, fix in business hours |

### Communicate

```
# Post to incident channel (adjust for your tool)
Slack: #incidents
"INCIDENT: [severity] — [brief description]. Investigating. Updates every 15 min."
```

## Phase 2: Diagnose (10-15 min)

### Check Recent Changes

```bash
# Was something just deployed?
vercel ls --limit 5
# or: kubectl rollout history deployment/myapp

# Recent commits
git log --oneline -10
```

### Check Database

```bash
# Active queries (look for long-running or blocked queries)
psql $PROD_DB_URL -c "
SELECT pid, now() - query_start AS duration, state, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER