runtime-verification

What this skill does

# Runtime Verification

Domain skill for verifying code works at runtime, beyond static analysis and unit tests.

## Iron Law

**NO COMPLETION UNTIL THE CODE BUILDS, RUNS, AND BEHAVES CORRECTLY. If you cannot verify it yourself, build the infrastructure to verify it.**

A green test suite is necessary but not sufficient. Runtime verification proves code actually works. "No test framework" is a problem to solve, not a reason to skip.

## Skip Whitelist (Enumerated — No Subjective Exemptions)

Runtime verification is **MANDATORY** for every change. The only permitted skips are the three categories below. Any skip outside this whitelist is forbidden and must be escalated via the Proactive-Autonomy protocol (see below).

| Skip Category | Definition | Required Evidence to Claim |
|---------------|------------|----------------------------|
| `markdown-only` | The diff touches **only** `.md`, `.markdown`, `.txt`, or `.rst` files. Zero code, config, or data files. | `git diff --name-only origin/$DEFAULT_BRANCH..HEAD` output showing only doc extensions. |
| `config-only` | The diff touches **only** configuration files (`.json`, `.yaml`, `.yml`, `.toml`, `.ini`, `.env.example`, dotfiles) with no executable code path changes. Config syntax must still be validated (lint/schema check, dry-run apply). | Full file list plus syntax validation output. |
| `dependency-bump-only` | The diff touches **only** lockfiles and manifest version strings (e.g., `package.json` version fields, `package-lock.json`, `poetry.lock`, `Gemfile.lock`, `go.sum`, `Cargo.lock`) with no source code, no config semantics, and no new dependencies. Build must still succeed. | Full file list plus successful build output. |

**If the diff mixes any whitelisted category with anything else (a single `.py` or `.ts` file, a new dependency, a config value change that alters behavior), the skip is disallowed. Run full runtime verification.**

### If In Doubt, Run It

**If you are uncertain whether the change qualifies for a whitelist skip — run runtime verification.** Uncertainty is never a reason to skip. The cost of an extra verification run is small; the cost of shipping unverified code is large.

Forbidden reasoning patterns include "small change", "CI-only edit", "tests already cover this", "I read the diff and it looks safe", "just a refactor", and similar. None of those are whitelist categories. If your reasoning does not map cleanly to `markdown-only`, `config-only`, or `dependency-bump-only` with the explicit evidence shown in the table above, you MUST run verification.

### Escalation Protocol for Out-of-Whitelist Skips

If you genuinely believe a skip outside the whitelist is warranted (e.g., infrastructure-only change, generated-code-only change, or something the whitelist does not yet cover), you MUST NOT proceed silently. Raise a Proactive-Autonomy escalation per [`references/escalation-format.md`](../../references/escalation-format.md) using `AskUserQuestion` and receive an explicit approval before proceeding. The Situation field MUST cite the specific change and why the whitelist does not cover it; the Tried field MUST list the standard paths attempted (fast-path verify script, build, smoke tests). Blanket "always skip for this repo" authorization is never valid — each out-of-whitelist skip requires its own escalation.

## Fast-Path Verification

Check for a project-level verify script first:

```bash
[ -x "verify.sh" ] && echo "FAST_PATH: verify.sh found"
[ -x "scripts/verify.sh" ] && echo "FAST_PATH: scripts/verify.sh found"
```

If found, run it and return results. Skip remaining steps.

## Build Verification

**Mandatory build step for all project types.** Build failure IS the finding — do NOT skip to runtime checks.

```bash
# Node.js / TypeScript
[ -f "package.json" ] && npm run build 2>&1

# Python
[ -f "setup.py" ] || [ -f "pyproject.toml" ] && pip install -e . 2>&1

# Go
[ -f "go.mod" ] && go build ./... 2>&1

# Rust
[ -f "Cargo.toml" ] && cargo build 2>&1

# Ruby
[ -f "Gemfile" ] && bundle install 2>&1
```

If build fails → iterate: read errors, fix, rebuild (up to `closedLoop.maxBuildIterations`, default 5). Do NOT proceed until the build passes.

## LSP Diagnostics Verification

**Pre-check**: Read `lsp.enabled` from settings (default `true`). If `false`, skip this section entirely.

When LSP is available and `lsp.diagnosticsAsQuality` is enabled in settings, collect language server diagnostics as an additional quality signal. This complements — never replaces — CLI-based quality commands.

### Process

1. Identify all files changed on the branch:
   ```bash
   git diff --name-only origin/$DEFAULT_BRANCH..HEAD
   ```

2. For each changed source file, use `LSP(documentSymbol)` to confirm the language server recognizes the file, then collect any diagnostics reported.

3. Map diagnostics to findings:

   | LSP Severity | Finding Priority | Action |
   |-------------|-----------------|--------|
   | Error | P1 | Must fix before proceeding |
   | Warning | P2 | Should fix |
   | Information/Hint | P3 | Consider |

4. Deduplicate against CLI tool output — if the same issue is reported by both LSP and a CLI tool (e.g., `tsc` and TypeScript LSP), keep only one entry.

### Timeout Handling

Each LSP operation must complete within `lsp.timeout` (default 5000ms from settings). If an operation times out:
- Mark that file's LSP check as "Timeout — skipped"
- Continue with remaining files
- Note timeout in output table
- Never block the workflow on a slow LSP server

### Graceful Fallback

If no LSP server is available, skip this section entirely with note: "LSP diagnostics: N/A — no language server configured." This is not an error and does not affect the verification outcome.

## Ad-Hoc Verification

For projects without formal test frameworks, verify by running the code:

| Project Type | Verification Approach |
|-------------|----------------------|
| Backend/API | Start server, curl endpoints, verify responses, check logs |
| CLI tools | Build, run with --help, run with sample input, check exit codes |
| Libraries | Write temporary test script, exercise public API, verify outputs, delete script |
| Static sites | Build, serve locally, verify pages load |
| Config-only | Validate config syntax, apply dry-run if supported |

"No test framework" is a problem to solve, not a reason to skip verification.

## Iterative Debug Loop

When any verification fails:
1. Read the FULL error message and stack trace — don't skim
2. Identify root cause (not just the symptom)
3. Fix the root cause
4. Re-verify

If the same error persists after a fix attempt: re-read code paths, try a different approach. Max `closedLoop.maxDebugIterations` (default 5) iterations, then escalate to user.

**The user should NEVER have to provide logs or tell you what went wrong.** You have access to the same errors — read them yourself.

## Discovery (dev server, port, E2E framework)

The `capability-discovery` skill (invoked at the start of every verify-relevant command) already detects tech stack, dev server scripts, and E2E frameworks. Re-running discovery here is redundant; consume the existing output.

When operating standalone (no prior `capability-discovery` invocation), use these probes:

```bash
# Dev server: CLAUDE.md hints, package.json scripts, framework config files
[ -f ".claude/CLAUDE.md" ] && grep -iE "(dev|server|start|serve):" .claude/CLAUDE.md
[ -f "package.json" ] && python3 -c "import json; d=json.load(open('package.json')); [print(f'{k}: {v}') for k,v in d.get('scripts',{}).items() if k in ('dev','start','serve')]"

# Port: running listeners on common ports
lsof -i -P -n 2>/dev/null | grep LISTEN | grep -E ':(3000|4000|5000|8000|8080)' | head -5

# E2E framework: config files
[ -f "playwright.config.ts" ] || [ -f "playwright.config.js" ] && echo "Playwright"
[ -f "cypress.config.ts" ] || [ -f "cypress.config.js" ] && echo "Cypress"
```

## Smoke Tests

If a dev server is running, perform basic hea