runbook-creation

What this skill does

# Runbook Creation Skill

## When to Use This Skill

Use this skill when:

- **Runbook Creation tasks** - Working on operational runbook templates for incident response and procedures
- **Planning or design** - Need guidance on Runbook Creation approaches
- **Best practices** - Want to follow established patterns and standards

## Overview

Create operational runbooks for incident response, maintenance procedures, and operational tasks.

## MANDATORY: Documentation-First Approach

Before creating runbooks:

1. **Invoke `docs-management` skill** for runbook patterns
2. **Verify SRE best practices** via MCP servers (perplexity)
3. **Base guidance on Google SRE principles**

## Runbook Types

```text
Runbook Categories:

┌─────────────────────────────────────────────────────────────────────────────┐
│  Incident Response Runbooks                                                  │
│  • Alert-triggered procedures                                                │
│  • Escalation paths                                                          │
│  • Communication templates                                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│  Operational Runbooks                                                        │
│  • Deployment procedures                                                     │
│  • Maintenance tasks                                                         │
│  • Backup/restore operations                                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│  Troubleshooting Runbooks                                                    │
│  • Diagnostic procedures                                                     │
│  • Common issue resolution                                                   │
│  • Debug workflows                                                           │
├─────────────────────────────────────────────────────────────────────────────┤
│  Emergency Runbooks                                                          │
│  • Disaster recovery                                                         │
│  • Security incident response                                                │
│  • Business continuity                                                       │
└─────────────────────────────────────────────────────────────────────────────┘
```

## Standard Runbook Template

```markdown
# Runbook: [TITLE]

| Property | Value |
|----------|-------|
| **ID** | RB-[NUMBER] |
| **Category** | [Incident/Operational/Troubleshooting/Emergency] |
| **Service** | [Service Name] |
| **Owner** | [Team/Individual] |
| **Last Updated** | [YYYY-MM-DD] |
| **Last Tested** | [YYYY-MM-DD] |
| **Review Frequency** | [Quarterly/Monthly/Annually] |

---

## Overview

**Purpose:** [What this runbook helps you accomplish]

**When to Use:** [Conditions that trigger this runbook]

**Expected Outcome:** [What success looks like]

**Estimated Duration:** [Time to complete]

---

## Prerequisites

### Required Access

- [ ] [System/Tool 1] - [Role/Permission needed]
- [ ] [System/Tool 2] - [Role/Permission needed]

### Required Knowledge

- [Skill/Knowledge 1]
- [Skill/Knowledge 2]

### Tools Needed

| Tool | Purpose | Access URL |
|------|---------|------------|
| [Tool 1] | [Purpose] | [URL/Link] |
| [Tool 2] | [Purpose] | [URL/Link] |

---

## Quick Reference

```text
Quick Commands:
┌────────────────────────────────────────────────────────────────┐
│ Check service status: kubectl get pods -n [namespace]          │
│ View logs: kubectl logs -f [pod-name] -n [namespace]           │
│ Restart service: kubectl rollout restart deployment/[name]     │
│ Check metrics: [monitoring-url]                                │
└────────────────────────────────────────────────────────────────┘
```

---

## Procedure

### Step 1: [Step Name]

**Objective:** [What this step accomplishes]

**Actions:**

1. [Action 1]

   ```bash
   # Command example
   kubectl get pods -n production
   ```

2. [Action 2]

**Expected Result:** [What you should see]

**If This Fails:** Go to [Troubleshooting Section](#troubleshooting)

---

### Step 2: [Step Name]

**Objective:** [What this step accomplishes]

**Actions:**

1. [Action 1]
2. [Action 2]

**Decision Point:**

```text
┌─────────────────────────────────────┐
│ Is the service responding?          │
│                                     │
│ YES → Continue to Step 3            │
│ NO  → Go to Step 4 (Escalation)     │
└─────────────────────────────────────┘
```

---

### Step 3: [Verification]

**Objective:** Verify the issue is resolved

**Verification Checklist:**

- [ ] Service is responding to health checks
- [ ] Metrics show normal values
- [ ] No new errors in logs
- [ ] Users can access the service

---

## Troubleshooting

### Issue: [Common Issue 1]

**Symptoms:** [What you observe]

**Cause:** [Root cause]

**Resolution:**

1. [Step 1]
2. [Step 2]

### Issue: [Common Issue 2]

**Symptoms:** [What you observe]

**Cause:** [Root cause]

**Resolution:**

1. [Step 1]
2. [Step 2]

---

## Escalation

### When to Escalate

- [ ] Issue not resolved after [X] minutes
- [ ] Impact affects [threshold]
- [ ] Required access not available
- [ ] Unsure of next steps

### Escalation Path

| Level | Contact | Method | Response Time |
|-------|---------|--------|---------------|
| L1 | On-call Engineer | PagerDuty | 15 min |
| L2 | Team Lead | Slack #incidents | 30 min |
| L3 | Engineering Manager | Phone | 1 hour |
| L4 | VP Engineering | Phone | As needed |

---

## Communication

### Status Updates

**Template:**

```text
[TIMESTAMP] - [SERVICE] - [STATUS]

Current Status: [Investigating/Identified/Monitoring/Resolved]
Impact: [Description of user impact]
Next Update: [Time of next update]

Actions Taken:
- [Action 1]
- [Action 2]

Next Steps:
- [Planned action]
```

### Stakeholder Notification

| Stakeholder | When to Notify | Method |
|-------------|----------------|--------|
| Engineering | Immediately | Slack |
| Product | If user-impacting | Slack |
| Support | If customer-facing | Email |
| Leadership | If SEV1/SEV2 | Phone |

---

## Post-Incident

### Cleanup Tasks

- [ ] Remove any temporary fixes
- [ ] Update monitoring/alerts if needed
- [ ] Document any new learnings

### Post-Incident Review

- [ ] Schedule post-mortem meeting
- [ ] Gather timeline and evidence
- [ ] Identify action items

---

## Appendix

### Related Runbooks

- [RB-XXX: Related Runbook 1]
- [RB-YYY: Related Runbook 2]

### Reference Documentation

- [Link to architecture docs]
- [Link to service docs]

### Revision History

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | [Date] | [Name] | Initial version |
| 1.1 | [Date] | [Name] | [Changes] |

```text

```

## Incident Response Runbook Template

```markdown
# Incident Runbook: [Alert Name]

| Property | Value |
|----------|-------|
| **Alert** | [Alert Name/ID] |
| **Severity** | [SEV1/SEV2/SEV3/SEV4] |
| **Service** | [Service Name] |
| **SLO Impact** | [Which SLO is affected] |

---

## Alert Details

**Trigger Condition:**
```text

[Alert query/condition]
Example: error_rate > 1% for 5 minutes

```

**Alert Meaning:** [What this alert indicates]

**False Positive Indicators:** [Signs this might be a false alarm]

---

## Immediate Actions (First 5 Minutes)

### 1. Acknowledge Alert

```bash
# Acknowledge in PagerDuty
pd incident:acknowledge

# Or via Slack
/pd ack
```

### 2. Assess Impact

**Quick Health Checks:**

```bash
# Check service status
curl -s https://api.example.com/health | jq .

# Check error rate
kubectl logs -l app=service --tail=100 | grep -c ERROR

# Check pod status
kubectl get pods -n production -l app=service
```

**Impact Assessment:**

| Check | Command | Expected | Actual |
|-------|---------|----------|--------|
| Health endpoint | `curl /health` | 200 OK | [Result] |
| Error rate | `grep ERROR` | < 10 | [Result] |