doc-claim-validator

What this skill does

# Documentation Claim Validator

Verify that what documentation *says* is actually *true* by extracting testable claims
and checking them against the codebase. Complements `doc-maintenance` (which handles
structural health) by handling **semantic accuracy**.

## When to Use

- After significant code changes (refactors, renames, API changes)
- Before releases — catch docs that describe removed or changed behavior
- When onboarding devs report "the docs are wrong"
- As a periodic trust audit on project documentation
- After running `doc-maintenance` to go deeper than structural checks

## Quick Reference

| Resource | Purpose | Load when |
|----------|---------|-----------|
| `scripts/extract_claims.py` | Deterministic claim extraction from markdown | Always (Phase 1) |
| `scripts/verify_claims.py` | Automated verification against codebase | Always (Phase 2) |
| `references/claim-taxonomy.md` | Full taxonomy of claim types with examples | Triaging unclear claims |

---

## Workflow Overview

```
Phase 1: Extract    → Pull verifiable claims from docs (deterministic script)
Phase 2: Verify     → Check claims against codebase (automated + AI)
Phase 3: Report     → Classify failures by severity and type
Phase 4: Remediate  → Fix or flag broken claims
```

---

## Phase 1: Extract Claims

Run the extraction script to parse all markdown files and pull out verifiable assertions:

```bash
python3 skills/doc-claim-validator/scripts/extract_claims.py [--json] [--root PATH] [--scope docs|manual|all]
```

The script extracts these claim types from markdown:

| Type | What it captures | Example in docs |
|------|-----------------|-----------------|
| `file_path` | Inline code matching file path patterns | `` `src/auth/login.ts` `` |
| `command` | Code blocks or inline code with shell commands | `` `npm run build` `` |
| `code_ref` | Function, class, method references in inline code | `` `authenticate()` `` |
| `import` | Import/require statements in code blocks | `import { Router } from 'express'` |
| `config` | Configuration keys, env vars, settings | `` `MAX_RETRIES=3` `` |
| `url` | External links (http/https) | `[docs](https://example.com)` |
| `architectural` | Verb-anchored prose claims about technology, integrations, or architectural patterns | "Uses Redis for caching", "follows the actor model", "delegated to Auth0" |
| `dependency` | Package/library name claims | "Uses Redis for caching" |
| `behavioral` | Assertions about what code does | "The system retries 3 times" |

The first 7 types are extracted deterministically. The script uses verb-anchored
regex for `architectural` (rules like `uses X`, `built with X`, `follows the X
pattern`, `delegated to X`, `via X`, `depends on X`) — this catches anchorless
prose claims that previously slipped through.

The last 2 (`dependency`, `behavioral`) require AI analysis and are handled in
Phase 2. `behavioral` in particular is *not* regex-extracted because behavioral
claims are free-form prose ("the cache invalidates when the user logs out")
that doesn't pattern-match cleanly — the behavioral verifier discovers and
verifies them in one pass.

**Output:** A structured list of claims with source file, line number, claim type, and the literal
text of the claim.

---

## Phase 2: Verify Claims

### Step 2a — Automated verification

Run the verification script on the extracted claims:

```bash
python3 skills/doc-claim-validator/scripts/verify_claims.py [--json] [--root PATH] [--claims-file PATH] [--check-staleness]
```

Pass `--check-staleness` to enable git-based drift analysis (see below).

The script checks each claim type differently:

| Claim type | Verification method | Pass condition |
|------------|-------------------|---------------|
| `file_path` | `os.path.exists()` | File exists at referenced path |
| `command` | `shutil.which()` + script check | Binary exists or script file exists |
| `code_ref` | `grep -r` for function/class name | Symbol found in codebase |
| `import` | Check module exists in project or deps | Module resolvable |
| `config` | Grep for config key in source | Key found in config files or code |
| `url` | HTTP HEAD request (optional, off by default) | Returns 2xx/3xx |

Pass `--check-urls` to enable URL verification (slow, requires network).

### Step 2b — AI-assisted verification

After the automated pass, dispatch agents to verify claims the script cannot.
Three of four verifiers run on `general-purpose` + `sonnet` — behavioral,
architectural, and code-example verification all require multi-file reasoning
that haiku's excerpt-read pattern strains under. The dependency verifier stays
on `Explore` + `haiku` because it's pure pattern matching against manifest
files.

#### Dispatch strategy: per-docfile batching

For behavioral and architectural verifiers, dispatch **one sonnet call per
markdown file containing claims of that type**, with all claims from that file
batched into a single prompt. This keeps each call's context budget on a small
number of related claims (cross-referencing within the doc improves
verification) while keeping total call count tied to doc-set size rather than
claim count. For a project with ~50 docs and ~150 architectural claims, expect
~10–20 sonnet calls (only docs with claims trigger calls), not 150.

For release audits where precision matters more than cost, run with
**per-claim dispatch** — one sonnet call per claim, each with the full doc as
context. Higher cost, higher precision.

#### Verifiers

**Verifier 1 — Dependency claim verifier** (`subagent_type: "Explore"`,
`model: "haiku"`):
Read `package.json`, `requirements.txt`, `go.mod`, `Cargo.toml`, or equivalent
dependency manifests. Cross-reference any doc claims about libraries,
frameworks, or services used. Report claims that reference dependencies not in
the project. Stays on haiku because pattern-matching against manifests doesn't
benefit from sonnet's reasoning.

**Verifier 2 — Behavioral claim verifier** (`subagent_type: "general-purpose"`,
`model: "sonnet"`, **per-docfile**):
For each markdown file in scope, dispatch a sonnet agent with the file content.
The agent (a) discovers behavioral claims in the file ("retries 3 times",
"caches for 5 minutes", "validates input before processing", "the cache
invalidates when the user logs out"), (b) finds the relevant code via grep /
codanna / `Read`, (c) verifies whether the claim matches the implementation.
Report each claim with confirmed / contradicted / unverifiable / conditional
status. Sonnet is needed because behavioral verification often requires
tracing across multiple files (handler → middleware → config) and
distinguishing happy-path from error-path behavior.

**Verifier 3 — Architectural claim verifier** (`subagent_type:
"general-purpose"`, `model: "sonnet"`, **per-docfile**):
For each markdown file with extracted `architectural` claims, dispatch a
sonnet agent with the file content and the list of pre-extracted claims. The
agent verifies each claim by:
- For `uses`/`built`/`depends`/`via` frames: check the named technology in
  dependency manifests, config files, and source imports.
- For `delegated` frames: check for SDK imports or HTTP integrations matching
  the named service.
- For `follows`/`uses_pattern` frames: check directory structure, class names,
  and code organization for the named architectural pattern (e.g., CQRS:
  separate command/query handlers + event store; hexagonal: adapters/ports
  dirs; saga: orchestrator class with named transitions).

Report each claim with confirmed / contradicted / unverifiable / conditional
status. Sonnet is needed because architectural patterns aren't 1:1 with any
single file — verification requires reading enough of the codebase to
recognize the pattern.

**Verifier 4 — Code example verifier** (`subagent_type: "general-purpose"`,
`model: "sonnet"`, **per-docfile**):
For code blocks in docs that show usage examples, verify the function
signatures, parameter names, return types, and import paths match