root-cause-tracing

What this skill does

<skill_overview>
Bugs manifest deep in the call stack; trace backward until you find the original trigger, then fix at source, not where error appears.
</skill_overview>

<rigidity_level>
MEDIUM FREEDOM - Follow the backward tracing process strictly, but adapt instrumentation and debugging techniques to your language and tools.
</rigidity_level>

<quick_reference>
| Step | Action | Question |
|------|--------|----------|
| 1 | Read error completely | What failed and where? |
| 2 | Find immediate cause | What code directly threw this? |
| 3 | Trace backward one level | What called this code? |
| 4 | Keep tracing up stack | What called that? |
| 5 | Find where bad data originated | Where was invalid value created? |
| 6 | Fix at source | Address root cause |
| 7 | Add defense at each layer | Validate assumptions as backup |

**Core rule:** Never fix just where error appears. Fix where problem originates.
</quick_reference>

<when_to_use>
- Error happens deep in execution (not at entry point)
- Stack trace shows long call chain
- Unclear where invalid data originated
- Need to find which test/code triggers problem
- Error message points to utility/library code

**Example symptoms:**
- "Database rejects empty string" ← Where did empty string come from?
- "File not found: ''" ← Why is path empty?
- "Invalid argument to function" ← Who passed invalid argument?
- "Null pointer dereference" ← What should have been initialized?
</when_to_use>

<the_process>
## 1. Observe the Symptom

Read the complete error:

```
Error: Invalid email format: ""
  at validateEmail (validator.ts:42)
  at UserService.create (user-service.ts:18)
  at ApiHandler.createUser (api-handler.ts:67)
  at HttpServer.handleRequest (server.ts:123)
  at TestCase.test_create_user (user.test.ts:10)
```

**Symptom:** Email validation fails on empty string
**Location:** Deep in validator utility

**DON'T fix here yet.** This might be symptom, not source.

---

## 2. Find Immediate Cause

What code directly causes this?

```typescript
// validator.ts:42
function validateEmail(email: string): boolean {
  if (!email) throw new Error(`Invalid email format: "${email}"`);
  return EMAIL_REGEX.test(email);
}
```

**Question:** Why is email empty? Keep tracing.

---

## 3. Trace Backward: What Called This?

Use stack trace:

```typescript
// user-service.ts:18
create(request: UserRequest): User {
  validateEmail(request.email); // Called with request.email = ""
  // ...
}
```

**Question:** Why is `request.email` empty? Keep tracing.

---

## 4. Keep Tracing Up the Stack

```typescript
// api-handler.ts:67
async createUser(req: Request): Promise<Response> {
  const userRequest = {
    name: req.body.name,
    email: req.body.email || "", // ← FOUND IT!
  };
  return this.userService.create(userRequest);
}
```

**Root cause found:** API handler provides default empty string when email missing.

---

## 5. Identify the Pattern

**Why empty string as default?**
- Misguided "safety": Thought empty string better than undefined
- Should reject invalid request at API boundary
- Downstream code assumes data already validated

---

## 6. Fix at Source

```typescript
// api-handler.ts (SOURCE FIX)
async createUser(req: Request): Promise<Response> {
  if (!req.body.email) {
    return Response.badRequest("Email is required");
  }
  const userRequest = {
    name: req.body.name,
    email: req.body.email, // No default, already validated
  };
  return this.userService.create(userRequest);
}
```

---

## 7. Add Defense in Depth

After fixing source, add validation at each layer as backup:

```typescript
// Layer 1: API - Reject invalid input (PRIMARY FIX)
if (!req.body.email) return Response.badRequest("Email required");

// Layer 2: Service - Validate assumptions
assert(request.email, "email must be present");

// Layer 3: Utility - Defensive check
if (!email) throw new Error("invariant violated: email empty");
```

**Primary fix at source. Defense is backup, not replacement.**
</the_process>

<debugging_approaches>
## Option 1: Guide User Through Debugger

**IMPORTANT:** Claude cannot run interactive debuggers. Guide user through debugger commands.

```
"Let's use lldb to trace backward through the call stack.

Please run these commands:
  lldb target/debug/myapp
  (lldb) breakpoint set --file validator.rs --line 42
  (lldb) run

When breakpoint hits:
  (lldb) frame variable email     # Check value here
  (lldb) bt                       # See full call stack
  (lldb) up                       # Move to caller
  (lldb) frame variable request   # Check values in caller
  (lldb) up                       # Move up again
  (lldb) frame variable           # Where empty string created?

Please share:
  1. Value of 'email' at validator.rs:42
  2. Value of 'request.email' in user_service.rs
  3. Value of 'req.body.email' in api_handler.rs
  4. Where does empty string first appear?"
```

---

## Option 2: Add Instrumentation (Claude CAN Do This)

When debugger not available or issue intermittent:

```rust
// Add at error location
fn validate_email(email: &str) -> Result<()> {
    eprintln!("DEBUG validate_email called:");
    eprintln!("  email: {:?}", email);
    eprintln!("  backtrace: {}", std::backtrace::Backtrace::capture());

    if email.is_empty() {
        return Err(Error::InvalidEmail);
    }
    // ...
}
```

**Critical:** Use `eprintln!()` or `console.error()` in tests (not logger - may be suppressed).

**Run and analyze:**

```bash
cargo test 2>&1 | grep "DEBUG validate_email" -A 10
```

Look for:
- Test file names in backtraces
- Line numbers triggering the call
- Patterns (same test? same parameter?)
</debugging_approaches>

<finding_polluting_tests>
## Finding Which Test Pollutes

When something appears during tests but you don't know which:

**Binary search approach:**

```bash
# Run half the tests
npm test tests/first-half/*.test.ts
# Pollution appears? Yes → in first half, No → second half

# Subdivide
npm test tests/first-quarter/*.test.ts

# Continue until specific file
npm test tests/auth/login.test.ts  ← Found it!
```

**Or test isolation:**

```bash
# Run tests one at a time
for test in tests/**/*.test.ts; do
  echo "Testing: $test"
  npm test "$test"
  if [ -d .git ]; then
    echo "FOUND POLLUTER: $test"
    break
  fi
done
```
</finding_polluting_tests>

<examples>
<example>
<scenario>Developer fixes symptom, not source</scenario>

<code>
# Error appears in git utility:
fn git_init(directory: &str) {
    Command::new("git")
        .arg("init")
        .current_dir(directory)
        .run()
}

# Error: "Invalid argument: empty directory"

# Developer adds validation at symptom:
fn git_init(directory: &str) {
    if directory.is_empty() {
        panic!("Directory cannot be empty"); // Band-aid
    }
    Command::new("git").arg("init").current_dir(directory).run()
}
</code>

<why_it_fails>
- Fixes symptom, not source (where empty string created)
- Same bug will appear elsewhere directory is used
- Doesn't explain WHY directory was empty
- Future code might make same mistake
- Band-aid hides the real problem
</why_it_fails>

<correction>
**Trace backward:**

1. git_init called with directory=""
2. WorkspaceManager.init(projectDir="")
3. Session.create(projectDir="")
4. Test: Project.create(context.tempDir)
5. **SOURCE:** context.tempDir="" (accessed before beforeEach!)

**Fix at source:**

```typescript
function setupTest() {
  let _tempDir: string | undefined;

  return {
    beforeEach() {
      _tempDir = makeTempDir();
    },
    get tempDir(): string {
      if (!_tempDir) {
        throw new Error("tempDir accessed before beforeEach!");
      }
      return _tempDir;
    }
  };
}
```

**What you gain:**
- Fixes actual bug (test timing issue)
- Prevents same mistake elsewhere
- Clear error at source, not deep in stack
- No empty strings propagating through system
</correction>
</example>

<example>
<scenario>Developer stops tracing too early</scenario>

<code>
# Error in API handler
async cr