incident-response

What this skill does

# Incident Response

## When to Use

Activate this skill when:
- Production service is down or returning errors to users
- Error rate has spiked beyond normal thresholds
- Performance has degraded significantly (latency increase, timeouts)
- An alert has fired from the monitoring system
- Users are reporting issues that indicate a systemic problem
- A failed deployment needs investigation and remediation
- Conducting a post-mortem or root cause analysis after an incident

**Output:** Write runbooks to `docs/runbooks/<service>-runbook.md` and post-mortems to `postmortem-YYYY-MM-DD.md`.

Do NOT use this skill for:
- Setting up monitoring or alerting rules (use `monitoring-setup`)
- Performing routine deployments (use `deployment-pipeline`)
- Docker image or infrastructure issues (use `docker-best-practices`)
- Feature development or code changes (use `python-backend-expert` or `react-frontend-expert`)

## Instructions

### Severity Classification

Classify every incident immediately. Severity determines response urgency, communication cadence, and escalation path.

| Severity | Impact | Examples | Response Time | Update Cadence |
|----------|--------|----------|---------------|----------------|
| **SEV1 (P1)** | Complete outage, all users affected | Service down, data loss, security breach | Immediate (< 5 min) | Every 15 min |
| **SEV2 (P2)** | Major degradation, most users affected | Core feature broken, severe latency | < 15 min | Every 30 min |
| **SEV3 (P3)** | Partial degradation, some users affected | Non-critical feature broken, intermittent errors | < 1 hour | Every 2 hours |
| **SEV4 (P4)** | Minor issue, few users affected | Cosmetic bug, edge case error | < 4 hours | Daily |

**Escalation rules:**
- SEV1: Page on-call engineer + engineering manager immediately
- SEV2: Page on-call engineer, notify engineering manager
- SEV3: Notify on-call engineer via Slack
- SEV4: Create ticket, address during normal working hours

See `references/escalation-contacts.md` for the contact matrix.

### 5-Minute Triage Workflow

When an incident is detected, follow this triage workflow within the first 5 minutes.

```
┌─────────────────────────────────────────────────────────┐
│  MINUTE 0-1: Acknowledge and Classify                   │
│  • Acknowledge the alert or report                      │
│  • Assign severity (SEV1-SEV4)                          │
│  • Designate incident commander                         │
├─────────────────────────────────────────────────────────┤
│  MINUTE 1-2: Assess Scope                               │
│  • Check health endpoints for all services              │
│  • Check error rate and latency dashboards              │
│  • Determine: which services are affected?              │
├─────────────────────────────────────────────────────────┤
│  MINUTE 2-3: Identify Recent Changes                    │
│  • Check: was there a recent deployment?                │
│  • Check: any infrastructure changes?                   │
│  • Check: any external dependency issues?               │
├─────────────────────────────────────────────────────────┤
│  MINUTE 3-4: Initial Communication                      │
│  • Post in #incidents channel                           │
│  • Update status page if SEV1/SEV2                      │
│  • Page additional responders if needed                 │
├─────────────────────────────────────────────────────────┤
│  MINUTE 4-5: Begin Investigation or Mitigate            │
│  • If recent deploy: consider immediate rollback        │
│  • If not deploy-related: begin diagnostic commands     │
│  • Start incident timeline log                          │
└─────────────────────────────────────────────────────────┘
```

**Quick health check command:**
```bash
./skills/incident-response/scripts/health-check-all-services.sh \
  --output-dir ./incident-triage/
```

### Incident Commander Role

The incident commander (IC) coordinates the response. They do NOT investigate directly.

**IC responsibilities:**
1. **Coordinate** -- Assign tasks to responders, prevent duplicate work
2. **Communicate** -- Post regular updates to stakeholders
3. **Decide** -- Make go/no-go decisions on rollback, escalation, communication
4. **Track** -- Maintain the incident timeline
5. **Close** -- Declare the incident resolved and schedule the post-mortem

**IC communication template (initial):**
```
INCIDENT DECLARED: [Title]
Severity: [SEV1/SEV2/SEV3/SEV4]
Commander: [Name]
Start time: [UTC timestamp]
Impact: [What users are experiencing]
Status: Investigating
Next update: [Time]
```

**IC communication template (update):**
```
INCIDENT UPDATE: [Title]
Severity: [SEV level]
Duration: [Time since start]
Status: [Investigating/Identified/Mitigating/Resolved]
Current findings: [What we know]
Actions in progress: [What we are doing]
Next update: [Time]
```

### Investigation Steps

Follow these diagnostic steps based on the type of issue.

#### Application Errors (FastAPI)

```bash
# 1. Check application logs for errors
./skills/incident-response/scripts/fetch-logs.sh \
  --service backend \
  --since "15 minutes ago" \
  --output-dir ./incident-logs/

# 2. Check error rate from logs
docker logs app-backend --since 15m 2>&1 | grep -c "ERROR"

# 3. Check active connections and request patterns
curl -s http://localhost:8000/health/ready | jq .

# 4. Check if the issue is in a specific endpoint
docker logs app-backend --since 15m 2>&1 | \
  grep "ERROR" | \
  grep -oP '"path":"[^"]*"' | sort | uniq -c | sort -rn

# 5. Check Python process status
docker exec app-backend ps aux
docker exec app-backend python -c "import sys; print(sys.version)"
```

#### Database Issues (PostgreSQL)

```bash
# 1. Check database connectivity
docker exec app-db pg_isready -U postgres

# 2. Check active connections (connection pool exhaustion?)
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT count(*), state FROM pg_stat_activity
  GROUP BY state ORDER BY count DESC;
"

# 3. Check for long-running queries (locks, deadlocks?)
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT pid, now() - pg_stat_activity.query_start AS duration,
         query, state
  FROM pg_stat_activity
  WHERE (now() - pg_stat_activity.query_start) > interval '30 seconds'
  AND state != 'idle'
  ORDER BY duration DESC;
"

# 4. Check for lock contention
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT blocked_locks.pid AS blocked_pid,
         blocking_locks.pid AS blocking_pid,
         blocked_activity.query AS blocked_query
  FROM pg_catalog.pg_locks blocked_locks
  JOIN pg_catalog.pg_stat_activity blocked_activity
    ON blocked_activity.pid = blocked_locks.pid
  JOIN pg_catalog.pg_locks blocking_locks
    ON blocking_locks.locktype = blocked_locks.locktype
    AND blocking_locks.relation = blocked_locks.relation
    AND blocking_locks.pid != blocked_locks.pid
  JOIN pg_catalog.pg_stat_activity blocking_activity
    ON blocking_activity.pid = blocking_locks.pid
  WHERE NOT blocked_locks.granted;
"

# 5. Check disk space
docker exec app-db df -h /var/lib/postgresql/data
```

#### Redis Issues

```bash
# 1. Check Redis connectivity
docker exec app-redis redis-cli ping

# 2. Check memory usage
docker exec app-redis redis-cli info memory | grep used_memory_human

# 3. Check connected clients
docker exec app-redis redis-cli info clients | grep connected_clients

# 4. Check slow log
docker exec app-redis redis-cli slowlog get 10

# 5. Check keyspace
docker exec app-redis redis-cli info keyspace
```

#### Network and Infrastructure

```bash
# 1. Check DNS resolution
nslookup api.example.com

# 2. Check SSL certificate expiry
echo | openssl s_client -servername api.example.com -connect api.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# 3. Check container resource usage
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# 4. Check disk space on host
df -h /

# 5. Check if dependent services are reachable
curl -sf htt