ds-verify

What this skill does

Announce: "Using ds-verify (Phase 5) to confirm reproducibility and completion."

## Contents

- [The Iron Law of DS Verification](#the-iron-law-of-ds-verification)
- [Verification Facts](#verification-facts)
- [The Verification Gate](#the-verification-gate)
- [Verification Checklist](#verification-checklist)
- [Reproducibility Demonstration](#reproducibility-demonstration)
- [Claims Requiring Evidence](#claims-requiring-evidence)
- [Insufficient Evidence](#insufficient-evidence)
- [Required Output Structure](#required-output-structure)
- [Completion Criteria](#completion-criteria)

## Context Monitoring

| Level | Remaining Context | Action |
|-------|------------------|--------|
| Normal | >35% | Proceed normally |
| Warning | 25-35% | Complete current review cycle, then trigger ds-handoff |
| Critical | ≤25% | Immediately trigger ds-handoff — do not start new review cycles |

# Verification Gate

Final verification with reproducibility checks and user acceptance interview.

<EXTREMELY-IMPORTANT>
## The Iron Law of DS Verification

**NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION. This is not negotiable.**

**Load shared enforcement first.**

Auto-load all constraints matching `applies-to: ds-verify`:

!`uv run python3 ${CLAUDE_SKILL_DIR}/../../scripts/load-constraints.py ds-verify`

**You MUST have these constraints loaded before proceeding. No claiming you "remember" them.**

Before claiming analysis is complete, you MUST:
1. RE-RUN - Execute analysis fresh (not cached results)
2. CHECK - Verify outputs match expectations
3. REPRODUCE - Confirm results are reproducible
4. ASK - Interview user about constraints and acceptance
5. Only THEN claim completion

This applies even when:
- "I just ran it"
- "Results look the same"
- "It should reproduce"
- "User seemed happy earlier"

**If you catch yourself thinking "I can skip verification," STOP — you're about to deliver unverified results that waste the user's time.**
</EXTREMELY-IMPORTANT>

## Verification Facts

- Review checks methodology; verify checks reproducibility. They are different gates — a passed review is not evidence the analysis reproduces.
- Reproducibility means re-running, not re-reading: prior results don't prove current reproducibility, because code, data, or environment may have changed since. Reading cached output proves nothing.
- Verifying your own work shares the implementer's biases — the reproducibility demonstration is run fresh, not vouched for by its author. "Verified" without re-execution is an unverified claim.
- The 10-minute reproducibility check is cheap against the 10 days of debugging when someone else can't run the analysis — skipping it to deliver faster is anti-efficient.
- "User will be happy" is an assumption, not acceptance — completion requires the user's explicit confirmation, not your prediction of it.

## Static Analysis (Constraint Check Scripts)

Before running runtime DQ checks, run the static analysis constraint check suite:

```bash
bash "${CLAUDE_SKILL_DIR}/../../scripts/check-all-ds.sh" "$(pwd)"
```

This runs all DS constraint check scripts (determinism, join audits, idempotency, error handling, schema contracts, standard errors, visualization integrity).

**If any check FAILS:** Report the failures in LEARNINGS.md. These are code quality issues in the analysis scripts that must be fixed before proceeding. Dispatch a fix subagent if needed.

**If all checks PASS:** Proceed to runtime DQ checks.

## The Verification Gate

**Checkpoint type:** decision (user confirms results — cannot auto-advance)

Before making ANY completion claim, follow this flowchart.

**This flowchart IS the specification. If prose elsewhere and this diagram disagree, the diagram wins.**

```
        ┌──────────────────────────────┐
        │ 1. RE-RUN (fresh, not cached) │
        └──────────────┬───────────────┘
                       ▼
        ┌──────────────────────────────┐
        │ 2. CHECK vs success criteria  │
        └──────────────┬───────────────┘
                  pass? │
            ┌───── no ──┴── yes ─────┐
            ▼                        ▼
   ┌─────────────────┐   ┌──────────────────────────┐
   │ NEEDS WORK →    │   │ 3. REPRODUCE             │
   │ log + dispatch  │   │ (same inputs→same outputs)│
   │ fix subagent    │   └────────────┬─────────────┘
   └────────┬────────┘        match?  │
            │           ┌──── no ──────┴── yes ───┐
            │           ▼                         ▼
            │  ┌─────────────────┐   ┌─────────────────────────┐
            │  │ NEEDS WORK →    │   │ 4. ASK — user           │
            │  │ non-determinism │   │ acceptance interview    │
            │  │ is a defect     │   └───────────┬─────────────┘
            │  └────────┬────────┘    accept?     │
            │           │      ┌── no/partial ────┴── yes ──┐
            │           │      ▼                            ▼
            │           │  ┌──────────────────┐  ┌────────────────────┐
            └───────────┴─▶│ loop: ds-fix /   │  │ 5. CLAIM COMPLETE  │
                           │ ds-implement,    │  │ (only after 1-4)   │
                           │ then re-verify   │  └────────────────────┘
                           └──────────────────┘
```

**Skipping any step is not verification.** Reaching step 5 without passing 1-4 is a false completion claim.

## Visual Diagnostics for Verification

When presenting verification results to the user in the acceptance interview, generate diagnostic plots to support the decision:

| Verification Check | Diagnostic to Generate |
|-------------------|----------------------|
| Reproducibility comparison | Overlay plot of Run 1 vs Run 2 key outputs |
| Data integrity | Pipeline waterfall chart (input rows → cleaning → joins → final) |
| Distribution sanity | Histogram/density plots of key variables with expected ranges annotated |
| Model performance | ROC curve, residual plot, or coefficient comparison (as appropriate) |

**Format:** Inline plots in notebooks, or saved to `scratch/diagnostics/` for script-based workflows. Present alongside the acceptance interview questions.

## Verification Checklist

### Technical Verification

#### Outputs Match Expectations
- [ ] All required outputs generated
- [ ] Output formats correct (files, figures, tables)
- [ ] Numbers are reasonable (sanity checks)
- [ ] Visualizations render correctly

#### Reproducibility Confirmed
- [ ] Ran analysis twice, got same results
- [ ] Random seeds produce consistent output
- [ ] No dependency on execution order
- [ ] Environment documented (packages, versions)

#### Data Integrity
- [ ] Input data unchanged
- [ ] Row counts traceable through pipeline
- [ ] No silent data loss or corruption

**Trace to Requirements:** For each success criterion, reference its requirement ID (e.g., "DATA-01: Panel has 50K+ firm-years — VERIFIED with df.shape output"). End-to-end traceability from SPEC.md through PLAN.md through VALIDATION.md through verification.

### User Acceptance Interview

**CRITICAL:** Before claiming completion, conduct user interview.

#### Step 1: Replication Constraints

```
AskUserQuestion:
  question: "Were there specific methodology requirements I should have followed?"
  options:
    - label: "Yes, replicating existing analysis"
      description: "Results should match a reference"
    - label: "Yes, required methodology"
      description: "Specific methods were mandated"
    - label: "No constraints"
      description: "Methodology was flexible"
```

If replicating:
- Ask for reference to compare against
- Verify results match within tolerance
- Document any deviations and reasons

#### Step 2: Results Verification

```
AskUserQuestion:
  question: "Do these results answer your original question?"
  options:
    - label: "Yes, fully"
      description: "Analysis addresses the core question"
    - label: "Partially"
      description: "Some aspects addressed, others missing"
    - label: "No"
      description: "Doe