agent-output-audit

What this skill does

# Agent Output Audit

Independent verification of AI-implemented work. The skill that asks: *"Did the implementing agent actually do what `task_NN.md` says it did?"* — not *"Would a real user succeed at this product?"* (that's `qa-execution`).

## Required Reading Router

Match your task to the row. Read the listed files **in full before** producing output. They are not appendices — they are load-bearing. Inline content in this SKILL.md is a pointer, not a substitute.

| Task                                                                 | MUST read                                                                                |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Discovering install/lint/test/build/start commands (Step 1)          | `references/project-signals.md`                                                          |
| Deciding E2E support and classifying coverage (Step 1)               | `references/e2e-coverage.md`                                                             |
| Building the audit scope checklist (Step 2)                          | `references/checklist.md`                                                                |
| Holding independent-evaluator stance on AI tasks (Step 3)            | `references/independent-evaluator-protocol.md`                                           |
| Scanning test diffs for AI hygiene red flags (Step 4)                | `references/ai-implementation-audit.md`                                                  |
| Diagnosing a test that passed on retry without a code change         | `references/flaky-triage.md`                                                             |

## Reference Index

- `references/project-signals.md` — Heuristics for picking install/lint/test/build/start commands across ecosystems when the repo lacks an umbrella gate.
- `references/e2e-coverage.md` — Taxonomy for `existing-e2e` / `needs-e2e` / `manual-only` / `blocked` and how to detect harness support.
- `references/checklist.md` — Audit checklist by category: contract discovery, baseline, task audit, AI hygiene, flaky detection, quality gates.
- `references/ai-implementation-audit.md` — Red Flag scanners (RF-1..RF-6), Requirement→Test mapping, verdict matrix for completed tasks.
- `references/independent-evaluator-protocol.md` — What counts (and doesn't count) as evidence; transcript classification (`genuine-failure` / `grader-bug` / `ambiguous-task` / `bypass-exploit`).
- `references/flaky-triage.md` — Taxonomy, diagnosis protocol, and quarantine workflow for retry-passes-without-code-change failures.

## Required Inputs

- **audit-output-path** (optional): Directory where audit artifacts (bugs, audit report, evidence) are stored. When provided, create the directory if it does not exist and use it for all audit outputs. When omitted, fall back to repository conventions or `/tmp/agent-output-audit-<slug>`.

## Procedures

**Step 1: Discover the Repository Verification Contract**

1. Read root instructions, repository docs, and CI/build files before running commands.
2. Execute `python3 scripts/discover-project-contract.py --root .` to surface candidate install, verify, build, test, lint, start commands, and E2E signals.
3. **STOP. Read `references/project-signals.md` in full before picking commands when discovery surfaces more than one plausible gate or the repo mixes ecosystems.**
4. **STOP. Read `references/e2e-coverage.md` in full before classifying any flow.**
5. Prefer repository-defined umbrella commands such as `make verify`, `just verify`, or CI entrypoints over language-default commands.
6. Resolve the audit artifact directory. If the user provided an `audit-output-path` argument, use it. Otherwise use repository conventions, falling back to `/tmp/agent-output-audit-<slug>`. Create the `audit/` subdirectory; store all bugs and reports under `<audit-output-path>/audit/`.
7. **Detect Compozy mode.** If `.compozy/tasks/<slug>/` exists, record the slug and switch into Compozy-aware audit:
   - Read `state.yaml` **(read-only — never write to it; `scripts/update-state.py` owns mutation per the cy-codex-loop contract)**.
   - Read `_techspec.md` (deliverable source of truth) and `_tasks.md` (task roster) when present.
   - List every `task_NN.md` and capture its frontmatter `status:` value (allowed: `pending`, `in_progress`, `completed`). When `task_NN.md` frontmatter disagrees with `state.yaml`, treat frontmatter as the source of truth.
   - Note the canonical memory slot `.compozy/tasks/<slug>/memory/qa-execution.md` — Step 4 writes audit notes there before any status flips.

**Step 2: Run the Baseline Verification Gate**

1. Install dependencies with the repository-preferred command.
2. Run the canonical verification gate once before any audit work. Execute in fastest-first order: lint and type-check, then build, then unit tests, then integration tests.
3. If the E2E command is separate from the umbrella gate, decide whether to run it now or after runtime prerequisites are ready, then record that plan explicitly.
4. If the baseline fails, read the first failing output carefully and determine whether it is pre-existing or introduced by current work before moving on.
4a. **Flaky-failure protocol.** When a baseline command fails, before classifying as pre-existing or new, run the failing test in isolation 3-5 times on the same SHA. If it passes at least once without code changes, classify as `flaky-suspect`, record in `audit-report.md` under `SUITE HEALTH SNAPSHOT` (test name, attempts, retry outcome, suspected category), and **do NOT promote to PASS via retry**. **STOP. Read `references/flaky-triage.md` in full before assigning a suspected category or proposing a quarantine.**

**Step 3: Audit Task Implementations (Compozy mode and any AI-implemented tasks)**

Skip this step only when no task, phase, PRD, tech spec, or implementation-plan artifacts exist.

1. **STOP. Read `references/independent-evaluator-protocol.md` in full before forming any task verdict.** Tripwire summary: never accept the implementing agent's transcript, success message, or memory note as evidence. In Compozy mode, read the implementing agent's `.compozy/tasks/<slug>/memory/<phase>.md` artifacts and classify anomalies (`genuine-failure` / `grader-bug` / `ambiguous-task` / `bypass-exploit`) in the `Errors / Corrections` section of `memory/qa-execution.md` **before** judging the task.
2. Read each `task_NN.md` and its body. Summarize each task into a Task Implementation Matrix (column names mirror cy-codex-loop frontmatter):
   - `task_path` (e.g., `.compozy/tasks/<slug>/task_07.md`)
   - `declared_status` — literal frontmatter `status:` value
   - `title`, `type`, `complexity`, `dependencies` — mirrored from frontmatter
   - `techspec_deliverable` — linked section in `_techspec.md` when present
   - Requirements, subtasks, checklist items, success criteria, dependent files
   - `implementation_evidence` — files, modules, routes, commands, migrations, seeds, tests
   - `verification_evidence` — commands executed, exit codes, output summaries
   - `qa_verdict` — `PASS` | `PARTIAL` | `FAIL` | `REOPEN` | `BLOCKED` (distinct from `declared_status`)
   - `ai_audit_findings` — red flag IDs that fired in Step 4 with verdict
   - `action` — `none` | `fixed` | `reopened-frontmatter` | `BUG-NNN.md filed`
   - `linked_bugs` — BUG IDs
3. Do not treat a task `declared_status`, checked checkbox, memory note, or prior agent summary as proof. Verify every completed or claimed-complete task against actual files, public behavior, automated tests, and acceptance criteria.
4. Classify each task with `qa_verdict`:
   - `PASS`: every material requirement and success criterion has implementation and fresh verification evidence.
   - `PARTIAL`: implementation exists but one or more non-critical requirements, tests, or evidence are missin