ds-validate

What this skill does

Announce: "Using ds-validate (Phase 3.5) to validate analysis outputs against SPEC.md requirements."

## Contents

- [The Iron Law of Validation](#the-iron-law-of-validation)
- [Validation Facts](#validation-facts)
- [Key Difference from Dev](#key-difference-from-dev)
- [The Process](#the-process)
- [Validation Levels](#validation-levels)
- [Classification](#classification)
- [VALIDATION.md Template](#validationmd-template)
- [Gate](#gate)
- [Phase Transition](#phase-transition)

# Output Validation Against SPEC.md

Phase 3.5 of the DS workflow (between implement and review). Maps every SPEC.md requirement to an output artifact and runs data quality checks.

<EXTREMELY-IMPORTANT>
## The Iron Law of Validation

**NO REVIEW WITHOUT VALIDATION. This is not negotiable.**

ds-review MUST NOT start until `.planning/VALIDATION.md` confirms all requirements have outputs. Validation is the DS equivalent of test coverage — without it, review is theater.
</EXTREMELY-IMPORTANT>

<EXTREMELY-IMPORTANT>
## Validation Facts

- Per-task checks during implement miss cross-task issues — joins that silently drop rows and filters that compound only surface in the end-to-end requirement-to-output mapping.
- LEARNINGS.md logs observations; VALIDATION.md maps requirements to outputs. They serve different purposes — one cannot substitute for the other.
- Row-count traceability cannot be eyeballed — "outputs look fine" is not a validation result, and review run on unvalidated outputs either misses the gaps or re-runs the same checks.
</EXTREMELY-IMPORTANT>

## Key Difference from Dev

DS validation does NOT auto-fill gaps. Dev's test-gap-auditor can write missing tests. DS gaps require human judgment — a wrong output means a wrong analysis, not just a missing test. When gaps are found, present them to the user and let the user decide: fix (return to implement) or accept (proceed to review).

## Static Analysis (Constraint Check Scripts)

Before running runtime DQ checks, run the static analysis constraint check suite:

```bash
bash "${CLAUDE_SKILL_DIR}/../../scripts/check-all-ds.sh" "$(pwd)"
```

This runs all DS constraint check scripts (determinism, join audits, idempotency, error handling, schema contracts, standard errors, visualization integrity).

**If any check FAILS:** Report the failures in LEARNINGS.md. These are code quality issues in the analysis scripts that must be fixed before proceeding. Dispatch a fix subagent if needed.

**If all checks PASS:** Proceed to runtime DQ checks.

## The Process

**This flowchart IS the specification. If prose elsewhere and this diagram disagree, the diagram wins.**

```
   ┌──────────────────────────────────────────────┐
   │ 0. RUN static analysis suite (check-all-ds.sh)│
   └───────────────────┬──────────────────────────┘
              all pass? │
        ┌──── no ───────┴────── yes ──────┐
        ▼                                  ▼
 ┌──────────────────┐   ┌───────────────────────────────────┐
 │ log to LEARNINGS │   │ 1-4. READ SPEC / PLAN / LEARNINGS, │
 │ + dispatch fix   │   │ DISCOVER ds-checks.md              │
 │ subagent, re-run │   └─────────────────┬─────────────────┘
 └────────┬─────────┘                     ▼
          │              ┌────────────────────────────────────┐
          │              │ 5. RUN ds-validate-coverage workflow│
          │              │ (one read-only validator/requirement│
          │              │  → JS gate, NOT a hand-tallied score)│
          │              └─────────────────┬──────────────────┘
          │                                ▼
          │              ┌────────────────────────────────────┐
          │              │ 6. RENDER .planning/VALIDATION.md   │
          │              │ from the workflow result            │
          │              └─────────────────┬──────────────────┘
          │                       JS gate   │
          │            ┌── gaps_found ───────┴── validated ──┐
          │            ▼                                     ▼
          │   ┌──────────────────────┐         ┌──────────────────────┐
          └──▶│ decision checkpoint: │         │ proceed to ds-review  │
              │ user fix-vs-accept   │         │ (gate: status=        │
              │ (see Gate section);  │         │  validated)           │
              │ accept ⇒ flip status │         └──────────────────────┘
              │ to validated         │
              └──────────────────────┘
```

> **Note:** Steps 1-4 stay in this skill as the reading/discovery preamble — the workflow's own Discover phase re-resolves them authoritatively, but reading them here lets the skill present context and decide scope before invoking the workflow.

### Step 1: Read Requirements

Read `.planning/SPEC.md` and extract every requirement:

```
For each requirement in SPEC.md:
  - Extract the requirement description
  - Note the success criteria
  - Note the expected output (table, figure, file, etc.)
```

### Step 2: Read Plan

Read `.planning/PLAN.md` and extract:
- Task-to-requirement mapping
- Output file locations mentioned
- Key columns and data structure decisions

### Step 3: Read Learnings

Read `.planning/LEARNINGS.md` and extract:
- Pipeline row counts at each stage (needed for DQ4 traceability)
- Data quality observations from implementation
- Any known issues or caveats

### Step 4: Load DQ Check Definitions

Read `${CLAUDE_SKILL_DIR}/../../skills/ds-implement/references/ds-checks.md` and follow its instructions.

### Step 5: Run the ds-validate-coverage workflow (per-requirement fan-out + JS gate)

The per-requirement DQ fan-out and the COVERED/PARTIAL/MISSING + `validated|gaps_found` gate are owned by a **ultracode workflow** — a script, not hand-dispatched agents. This is why: the validators return RAW DQ statuses and the **gate is computed in pure JS from those statuses**, so the model can no longer tally the composite by hand (the old honor-system gate). The workflow also isolates one validation transcript per requirement out of main context.

**1. Resolve the cached workflow path:**

```bash
WF=$(command ls -d ~/.claude/plugins/cache/edwinhu-plugins/workflows/*/workflows/ds-validate-coverage.js 2>/dev/null | sort -V | tail -1)
# Local-plugin fallback (running from source, cache empty):
[ -z "$WF" ] && WF="${CLAUDE_SKILL_DIR}/../../workflows/ds-validate-coverage.js"
echo "$WF"
```

**2. Run it** (full pass first; on a re-run after fixes, pass `onlyChecks` + `priorReviews` from the prior result):

```
Workflow({ scriptPath: "<WF>", args: { projectDir: "<abs project dir>", pluginRoot: "<abs .../workflows dir>" } })
```

The workflow fans out one **read-only** validator per in-scope SPEC requirement (running DQ1-DQ5 + M1 from `ds-checks.md`), then computes — in JS, from raw statuses — each requirement's classification and the overall `status`. It returns `{ overallPass, status, counts, scoreTable, findings, reviews, reviewersThatFlagged }`.

### Step 6: Render VALIDATION.md from the workflow result

**Do NOT recompute or rationalize the gate** — `result.status` and `result.overallPass` are computed in JS. Write `.planning/VALIDATION.md` using `result.scoreTable` as the Requirements Map, `result.counts` for the frontmatter totals, and `result.findings` under DQ Details:

```
status: <result.status>           # validated | gaps_found — verbatim from the workflow
requirements_total / covered / partial / missing: <result.counts>
Requirements Map: <result.scoreTable>
DQ Details: <result.findings>
```

**The `/goal` fix loop stays in this skill:** if `status: gaps_found`, present gaps (Step "Gate" below) and let the **user** decide fix vs accept. On a fix-and-re-validate cycle, re-run the workflow with `onlyChecks: <prev result.reviewersThatFlagged>` and `priorReviews: <prev result.reviews>` so unflagged requirements carry forward and only the gaps re-run live.

## Validation Levels

Each requirement is validated at four levels, in order:

| Level | Check | Example |
|-------|-------|-