post-mortem

What this skill does

# Post-Mortem (Blameless Incident Review)

## Overview

A post-mortem is a structured, blameless review held after an incident, outage, regression, missed launch, or failed experiment. The goal is not to assign fault but to learn how the system (people, process, code, and organization) produced the outcome, and to commit to durable changes that reduce the chance of recurrence.

This skill operationalizes the Google SRE blameless post-mortem template, the Etsy "morgue" tradition, John Allspaw's "How Complex Systems Fail" reading, Charles Perrow's Normal Accident Theory, and Sidney Dekker's *Field Guide to Understanding "Human Error"*. Where the companion `discovery/pre-mortem/` skill imagines failure before it happens, post-mortem learns from failure that already did.

### When to Use

- **Severity 1 / Severity 2 incident** — customer-facing outage, data loss, security event, payments failure, or significant regression.
- **Sev 3 with novelty** — even a small incident is worth a post-mortem if it surfaced a class of failure the team has not seen before.
- **Missed launch or rolled-back release** — the launch itself is the incident.
- **Failed experiment with a negative business outcome** — a pricing test that depressed revenue, an onboarding change that hurt activation.
- **Customer escalation** — an executive customer call where the product was the proximate cause.
- **Near miss** — the deploy that "almost" took the site down. Post-mortems on near misses are some of the highest-leverage learning a team gets.

If the incident is below the team's severity threshold and the team has seen the same class of failure recently, document the recurrence in the existing post-mortem rather than producing a new one.

## Severity Thresholds

| Severity | Customer impact | Post-mortem required? | Distribution |
|---|---|---|---|
| **Sev 0 / Sev 1** | Full outage, data loss, security incident | Mandatory; within 5 business days | All engineering + execs |
| **Sev 2** | Partial outage, major degradation, payments impact | Mandatory; within 7 business days | All engineering |
| **Sev 3** | Single-feature degradation, recoverable | Optional unless novel pattern | Owning team |
| **Sev 4** | Minor issue, manual workaround available | Skip unless near-miss | None |
| **Near miss** | Almost caused Sev 0/1/2 | Recommended | Owning team + SRE |

Severity is set at incident open and confirmed at incident close; it can be revised upward but rarely downward.

## Blameless Principles

The single most-cited reason post-mortems fail to produce learning is that participants feel unsafe. Blameless does not mean "consequence-free" — accountability still exists for following process. It means: assume that everyone involved acted reasonably given what they knew at the time, and focus on the system that surrounded their decision.

### Five blameless ground rules

1. **No names in the narrative.** Refer to roles, not people. "The on-call engineer", not "Sarah". Names appear only in the contacts table and the action-item owner column.
2. **No "should have".** Replace with "the system did not surface". "The on-call should have noticed the queue depth" → "The dashboard did not alert on queue depth above 10,000".
3. **No counterfactuals in the root cause.** "If only X had not happened" is not a cause; it is a wish. Stick to mechanisms.
4. **Human error is a symptom, not a cause** (Dekker). When a human acted "incorrectly", the goal is to understand why that action was the locally rational thing to do.
5. **Hindsight bias is the enemy.** What is obvious now was not obvious then. Reconstruct what was knowable in the moment, not what is knowable now.

### The Allspaw test

Adapted from John Allspaw's writing: after writing the post-mortem, ask: *"Would I send this document to the engineer who pushed the button, and would they feel that it represented their experience fairly?"* If no, rewrite until yes.

## Post-Mortem Template Sections

A standard post-mortem has 10 sections, in this order:

1. **Header** — title, date, severity, duration, status (draft / final), authors.
2. **Summary** — 3-5 sentences. What happened, who was affected, how it was resolved, what the durable fix is. A reader who reads only this section should still know the headline.
3. **Impact** — customers affected (count and segments), revenue impact, SLO error budget burned, internal teams blocked.
4. **Timeline** — minute-by-minute log of events from detection to resolution. Include the trigger, detection, escalation, mitigation, and resolution timestamps. Mark "moment of decision" rows.
5. **What went well** — actions, tooling, or decisions that reduced impact. This section is critical and is the one most often skipped. It anchors the document in learning, not just blame avoidance, and surfaces practices worth replicating.
6. **What went wrong** — the system conditions that allowed the incident to happen and to persist. Phrased as system properties, not human failures.
7. **Contributing factors** — the list of conditions that, in combination, produced the outcome. Distinguish from root cause (below).
8. **Root cause analysis** — 5 Whys or Causal Tree (see next section). Document the method used.
9. **Action items** — durable changes. Each item has an owner, a due date, a tracking issue ID, and a category (prevent / detect / mitigate / respond / process).
10. **Lessons learned** — broader patterns the team takes away. These often inform engineering standards, runbooks, or hiring.

## Root Cause Methods

### 5 Whys

A linear technique popularized by Toyota. Start with the proximate effect and ask "why?" five times. Each answer becomes the next question.

```
1. Why did the API return 500 errors for 22 minutes?
   → Because the database connection pool was exhausted.
2. Why was the connection pool exhausted?
   → Because a batch job opened 200 connections without releasing them.
3. Why did the batch job not release them?
   → Because the connection lifecycle is managed by a context manager that was bypassed on the error path.
4. Why was the error path bypassing the context manager?
   → Because a refactor in March introduced an early return before the `finally` block.
5. Why did the refactor land without catching this?
   → Because the test suite does not exercise the connection-pool exhaustion path.
```

**Strengths:** fast, anyone can run it, fits in a 15-minute slot.

**Limits:** linear — only finds one chain. Real incidents usually have multiple contributing factors that combine non-linearly. Use 5 Whys when the failure is a clear chain. Use Causal Tree when it is not.

### Causal Tree (Allspaw / SRE style)

A tree where the root is the incident outcome and the branches are the contributing factors. Each factor can have its own sub-factors. Unlike 5 Whys, the tree allows for parallel chains and AND/OR relationships.

```
Outcome: API returned 500s for 22 minutes
├── Connection pool exhausted
│   ├── Batch job leaked connections (code defect)
│   │   ├── Early-return bypassed context manager
│   │   └── Test suite did not cover error path
│   └── Pool size set conservatively for cost reasons
│       └── Capacity planning predates current traffic shape
├── Alert fired late
│   ├── Threshold set at pool 95% (no warn at 80%)
│   └── PagerDuty escalation path stale (rotated owner)
└── Mitigation took 12 minutes
    ├── Runbook missing pool-restart command
    └── New on-call had not shadowed a database incident
```

**Strengths:** captures the multi-factor nature of complex systems failures. Maps cleanly to action items (one mitigation per leaf or sub-tree).

**Limits:** longer to produce. Requires a facilitator who can keep the team from arguing the tree structure rather than the substance.

### Choosing between them

| Use 5 Whys when | Use Causal Tree when |
|---|---|
| Single clear chain of cause | Multiple contributing factors |
| Sev 3 / Sev 4 incidents | Sev 0 / Sev 1 / Sev 2 incidents |
| Pressed for time | Recurrin