investigate

What this skill does

# Systematic Debugging

## Iron Law

**NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST.**

Fixing symptoms creates whack-a-mole debugging. Every fix that doesn't address root cause makes the next bug harder to find. Find the root cause, then fix it.

---

## Step 0: Investigation Context Detection

Before investigating, detect what you're working with. The context determines which phases to emphasize.

**Auto-detect the investigation context:**

| Signal | Context type | Starting point |
|--------|-------------|----------------|
| User pasted a stack trace or error message | **Error report** | Parse the trace → jump to Phase 1 step 2 (read the code) |
| A test is failing (`test`, `spec`, `_test` in the message) | **Failing test** | Run the test → read the assertion → trace backwards |
| User says "it was working before" or references a recent change | **Regression** | `git log` / `git bisect` → find the breaking commit |
| User describes unexpected behavior (no error) | **Logic bug** | Reproduce → add assertions to narrow the gap between expected and actual |
| User reports slowness, timeouts, high resource usage | **Performance bug** | Jump to Performance Investigation Branch (below) |
| User says "works locally, fails in CI/staging/prod" | **Environment bug** | Jump to Environment Comparison Checklist (below) |
| Issue tracker ticket or TODO reference | **Tracked issue** | Read the ticket/TODO → gather prior context before investigating |

Use AskUserQuestion **only if** you cannot determine the context from the user's message and available project state. Ask ONE question:
```
I need a bit more context to investigate effectively.

A) I have an error message or stack trace to share
B) A test is failing — here's which one: [name]
C) Something that was working is now broken
D) The behavior is wrong but there's no error
E) It's a performance/slowness issue
F) It works locally but fails elsewhere
```

---

## Step 0A: Investigation Pacing

*Choose pacing based on bug complexity. Simple bugs shouldn't get the full treatment.*

**Quick-pass mode** — use when:
- The error message points directly to the cause (e.g., typo, missing import, undefined variable)
- A single test is failing with an obvious assertion mismatch
- The user already knows the root cause and just needs the fix verified

Quick-pass flow: Phase 1 (abbreviated) → Phase 4 → Phase 5. Skip Phases 2-3.

**Full investigation mode** — use when:
- The bug is intermittent or non-deterministic
- Multiple symptoms or multiple failing tests
- Prior fix attempts have failed
- The cause is not obvious from the error message
- Performance or environment bugs

If Quick-pass stalls after 10 minutes or one failed hypothesis, **escalate to Full investigation**.

---

## Step 0B: Evidence Log

*Track everything you check. Investigations fail when you forget what you've ruled out.*

Maintain a running evidence log throughout the investigation. Update it after every significant action:

```
EVIDENCE LOG
────────────────────────────────────────
#  | Phase | What was checked              | Finding              | Status
1  | P1    | Stack trace line 42           | NPE in user.name     | CONFIRMED
2  | P1    | git log -5 -- user.rb         | No recent changes    | RULED OUT
3  | P2    | Pattern: nil propagation       | Matches signature    | HYPOTHESIS
4  | P3    | Added assert at line 38       | user is nil          | CONFIRMED
────────────────────────────────────────
```

Statuses: **CONFIRMED** (evidence supports), **RULED OUT** (evidence contradicts), **HYPOTHESIS** (untested), **INCONCLUSIVE** (needs more data)

---

## Phase 1: Root Cause Investigation

*_5 Whys — don't stop at the first "because." Each answer should prompt "but why?"_*

Gather context before forming any hypothesis.

1. **Collect symptoms:** Read the error messages, stack traces, and reproduction steps. If the user hasn't provided enough context, ask ONE question at a time via AskUserQuestion.

2. **Read the code:** Trace the code path from the symptom back to potential causes. Use Grep to find all references, Read to understand the logic.

3. **Check recent changes:**
   ```bash
   git log --oneline -20 -- <affected-files>
   ```
   Was this working before? What changed? A regression means the root cause is in the diff.

   **If this is a regression**, use git bisect to find the breaking commit:
   ```bash
   git bisect start
   git bisect bad              # current state is broken
   git bisect good <known-good-ref>  # last known working state
   # Run reproduction at each step, mark good/bad
   ```
   *Git bisect finds the breaking commit in O(log n) steps. For a regression, this is almost always faster than reading diffs manually.*

4. **Check the data, not just the code:** *_Many bugs are data bugs, not code bugs._*
   - Query the database for the affected record(s) — is the data what you expect?
   - Check cache state (Redis keys, CDN headers, browser storage)
   - Check queue state (pending jobs, dead letter queues)
   - Check file system state (permissions, missing files, stale locks)
   - Check external API responses (has the contract changed?)

5. **Reproduce:** Can you trigger the bug deterministically? If not, gather more evidence before proceeding.

Update the Evidence Log after each step.

Output: **"Root cause hypothesis: ..."** — a specific, testable claim about what is wrong and why.

---

## Phase 2: Pattern Analysis

*_Occam's Razor — the simplest explanation that fits all the evidence is most likely correct. Resist exotic hypotheses when a common pattern matches._*

Check if this bug matches a known pattern:

| Pattern | Signature | Where to look |
|---------|-----------|---------------|
| Race condition | Intermittent, timing-dependent, passes alone / fails in suite | Concurrent access to shared state, async operations |
| Nil/null propagation | NoMethodError, TypeError, "undefined is not a function" | Missing guards on optional values, empty collections |
| State corruption | Inconsistent data, partial updates | Transactions, callbacks, hooks, shared mutable state |
| Integration failure | Timeout, unexpected response, connection refused | External API calls, service boundaries, network config |
| Configuration drift | Works locally, fails in staging/prod | Env vars, feature flags, DB state, secrets |
| Stale cache | Shows old data, fixes on cache clear | Redis, CDN, browser cache, OPcache, compiled assets |
| Off-by-one | Fence-post errors, missing last item, index out of bounds | Loops, pagination, array slicing, date ranges |
| Encoding/charset | Mojibake, truncated strings, failed comparisons | UTF-8 vs Latin-1, BOM markers, binary in text fields |
| Timezone/locale | Wrong dates, off-by-N-hours, failed date comparisons | TZ-unaware timestamps, locale-dependent formatting |
| Dependency version | "method not found" on valid method, changed behavior | Lock file drift, transitive dependency updates |
| Permission/ownership | "access denied", silent failures, empty results | File permissions, DB grants, IAM roles, CORS |
| Silent truncation | Data shorter than expected, no error | VARCHAR limits, integer overflow, payload size limits |
| Memory leak | Gradual slowdown, OOM after N requests | Unclosed connections, growing caches, event listener accumulation |
| Deadlock | Hangs indefinitely, no error, no timeout | Nested locks, DB row-level locks, circular waits |

Also check:
- `TODOS.md` for related known issues
- `git log` for prior fixes in the same area — **recurring bugs in the same files are an architectural smell**, not a coincidence

**External pattern search:** If the bug doesn't match a known pattern above, use WebSearch (or dispatch an Agent for parallel search) for:
- "{framework} {generic error type}" — **sanitize first:** strip hostnames, IPs, file paths, SQL, customer data. Search the error category, not the raw message.
- "{library} {component} known issues"

If WebSearch is unavailable, skip this sea