ds-fix

What this skill does

**Announce:** "Using ds-fix for mid-analysis course correction."

## Session Resume Detection

Before starting, check for an existing handoff:

1. Check if `.planning/HANDOFF.md` exists
2. **If found:** Read it and present to user:
   - Show the phase, task progress, and Next Action from the handoff
   - Ask: "Resume from handoff, or start fresh?"
   - If resume: skip to the recorded phase
   - If fresh: proceed with diagnosis
3. **If not found:** Proceed normally with Step 1 (Load Context)

## Where This Fits

```
/ds (entry) → brainstorm → plan → implement → review → verify
                                    ↑
/ds-fix (midpoint) ─────────────────┘
```

This is the re-entry point. Jump back into a DS workflow that needs fixing.

<EXTREMELY-IMPORTANT>
## The Iron Law of DS Editing

**DIAGNOSE BEFORE FIXING. This is not negotiable.**

Before changing ANY analysis code, you MUST:
1. Load the workflow context
2. Identify WHAT is wrong (specific, not vague)
3. Identify WHY it's wrong (root cause)
4. Only THEN fix it (with output-first verification)

**If you're about to change code without diagnosing first, STOP.**
</EXTREMELY-IMPORTANT>

### Fix Facts

- "The data probably changed" is a hypothesis, not a diagnosis — it is only established by comparing data profiles before/after: shape, dtypes, distributions, nulls.
- Parameter-tweaking until the error goes away is specification search applied to debugging. A fix whose mechanism you can't state is cosmetic — the bug returns in a different form.
- A traceback locates WHERE the pipeline failed, not WHY — the root cause is found by tracing backwards to the first divergence point, and if you don't know why it failed, a passing rerun tells you nothing about why it passed.

## Step 1: Load Context

Read workflow state, shared enforcement, AND shared check definitions:

As the midpoint, auto-load ALL constraints matching `applies-to: ds-fix` (midpoint can route to any phase):

!`uv run python3 ${CLAUDE_SKILL_DIR}/../../scripts/load-constraints.py ds-fix`

**You MUST have these constraints loaded before proceeding. No claiming you "remember" them.**

```
Read(".planning/SPEC.md")
Read(".planning/PLAN.md")
Read(".planning/LEARNINGS.md")
```

Read `${CLAUDE_SKILL_DIR}/../../skills/ds-implement/references/ds-checks.md` and follow its instructions.

**The shared checks file contains data quality check definitions (DQ1-DQ6, M1, R1) used by both ds-review and ds-fix.** Loading it here ensures the midpoint runs identical checks to the entry point's review phase. Without it, checks drift apart and the midpoint misses issues review would catch.

If no workflow state exists, suggest starting with `/ds` instead.

### SAS Project Detection

After loading PLAN.md, check if `Implementation Language` is `SAS` or `Mixed`. If so, reload SAS enforcement before any fix:

Read `${CLAUDE_SKILL_DIR}/../../skills/wrds/references/sas-etl.md` and follow its instructions.

**SAS projects have unique failure modes** (hash merge memory, WHERE function wrapping, SGE array misconfiguration). The SAS enforcement must be loaded BEFORE diagnosing — otherwise you will misdiagnose SAS-specific issues as generic bugs.

### Context Monitoring

Before starting diagnosis, check context availability:

| Level | Remaining Context | Action |
|-------|------------------|--------|
| Normal | >35% | Proceed with diagnosis and fix |
| Warning | 25-35% | Complete current fix, then invoke ds-handoff |
| Critical | ≤25% | Invoke ds-handoff immediately — no new fixes |

**At Warning level:** After current fix completes, invoke:
Read `${CLAUDE_SKILL_DIR}/../../skills/ds-handoff/SKILL.md` and follow its instructions.

**Why:** A multi-step fix pipeline with 20% context remaining produces degraded output. Better to handoff cleanly and resume fresh.

## Step 2: Diagnose

Identify the issue category:

| Category | Symptoms | Route To |
|----------|----------|----------|
| **Runtime Error** | Traceback, cell failure, import error | Debug Protocol |
| **Wrong Results** | Numbers don't match expectations, sanity checks fail | Re-analysis Protocol |
| **Unclear Root Cause** | 3+ plausible explanations, mysterious data quality issues | Competing Hypothesis Investigation |
| **Reviewer Feedback** | Specific methodology concerns, requested changes | Revision Protocol |
| **Data Change** | New data available, source updated, schema changed | Re-profiling Protocol |
| **Scope Change** | New questions, expanded requirements | Spec Update → re-plan |

Ask user if ambiguous:

```
AskUserQuestion(questions=[
  {
    "question": "What needs fixing in your analysis?",
    "header": "Issue type",
    "options": [
      {"label": "Runtime error", "description": "Code fails, traceback, import error"},
      {"label": "Wrong results", "description": "Numbers don't look right, sanity checks fail"},
      {"label": "Unclear root cause", "description": "Multiple plausible explanations, mysterious data quality"},
      {"label": "Reviewer feedback", "description": "Specific changes requested by reviewer"},
      {"label": "Data/scope change", "description": "New data, updated requirements, new questions"}
    ],
    "multiSelect": false
  }
])
```

### Diagnostic Routing Flowchart

```
┌───────────────────────┐
│  Load Context (Step 1) │
│  SPEC + PLAN + LEARN   │
└───────────┬───────────┘
            ▼
┌───────────────────────┐
│  Identify Symptoms     │
│  (Step 2)              │
└───────────┬───────────┘
            ▼
     ┌──────┴──────┐
     │  Traceback?  │─── YES ──→ Debug Protocol
     └──────┬──────┘              (Runtime Error)
            │ NO
            ▼
     ┌──────────────┐
     │ Numbers wrong │─── YES ──→ Re-analysis Protocol
     │ or unexpected?│              (Trace backwards)
     └──────┬───────┘
            │ NO
            ▼
     ┌──────────────┐
     │ 3+ plausible │─── YES ──→ Competing Hypothesis
     │ explanations?│              (Parallel investigation)
     └──────┬───────┘
            │ NO
            ▼
     ┌──────────────┐
     │ Reviewer      │─── YES ──→ Revision Protocol
     │ feedback?     │              (Fix per feedback)
     └──────┬───────┘
            │ NO
            ▼
     ┌──────────────┐
     │ Data or scope │─── YES ──→ Re-profiling / Spec Update
     │ changed?      │
     └──────────────┘
```

**This flowchart IS the diagnostic spec.** If the category table and flowchart disagree, the flowchart wins.

## Step 3: Fix by Category

### Runtime Error → Debug Protocol

1. Identify the failing cell/script
2. Read the error traceback
3. Check LEARNINGS.md for prior context on this step
4. Fix with output-first verification:
   - Print state BEFORE the failing operation
   - Fix the operation
   - Print state AFTER
   - Verify output matches expectations from PLAN.md

For notebook-specific errors, load notebook-debug patterns:

Read `${CLAUDE_SKILL_DIR}/../../skills/notebook-debug/SKILL.md` and follow its instructions.

### Wrong Results → Re-analysis Protocol

<EXTREMELY-IMPORTANT>
**Trace backwards. Do NOT guess forward.**

The bug is at the FIRST step where output diverges from expected. Find that step.
</EXTREMELY-IMPORTANT>

1. Identify WHICH results are wrong
2. Trace backwards through the pipeline:
   - Check final output → is the aggregation wrong?
   - Check intermediate data → is the transformation wrong?
   - Check input data → is the loading/filtering wrong?
3. Find the FIRST step where output diverges from expected
4. Fix that step with output-first verification
5. Re-run downstream steps and verify

### Reviewer Feedback → Revision Protocol

1. Document each piece of feedback as a task
2. For each task:
   - Locate the relevant code/output
   - Apply the change
   - Verify with output-first protocol
   - Document in LEARNINGS.md
3. After all changes, re-run review checks:

Read `${CLAUDE_SKILL_DIR}/../../skills/ds-review/SKILL.md` and follow its instructions.

### Data/Scope Change → Re-profiling Protocol

1. Profile the new data