evaluate-pipeline-output

What this skill does

# Pipeline Output Evaluator

Perform a comprehensive semantic quality evaluation of a contact-center-lab pipeline run. This skill reads pipeline source code to discover schemas and configuration, then reads every key output file, compares content against expectations derived from both the code and the input, and produces a structured report with severity-rated findings.

**This skill is read-only. It never modifies files, commits, or pushes.**

> [!WARNING]
> **Scope:** Specialized for the contact-center-lab pipeline. Not generalizable without substantial rewrite. See `LAB_NOTEBOOK.md` in that project for pipeline schema.

## Proactive Triggers

Suggest this skill when:
1. User completes a pipeline run and wants to evaluate quality
2. User asks "did that run produce good output?" or "evaluate the results"
3. Before deciding to run the full 10K corpus
4. After implementing any pipeline code change to verify quality is maintained

## Input

**Arguments:** `$ARGUMENTS`

Required:
- `<output-dir>` — path to the pipeline output directory (e.g., `./output/single-2026-03-05-1803`)

Optional:
- `--input <path>` — path to input fixture/data directory, used to read raw source articles for comparison
- `--baseline <previous-output-dir>` — path to a prior run's output directory for regression comparison
- `--mode test|validation|production` — adjusts severity thresholds (default: `validation`)

**Usage:**
```text
/evaluate-pipeline-output ./output/single-2026-03-05-1803
/evaluate-pipeline-output ./output/full-run-2026-03-06 --baseline ./output/full-run-2026-03-04
/evaluate-pipeline-output ./output/test-5 --input ./pipeline/tests/fixtures/five --mode test
```

If the output directory is not provided, ask the user which run to evaluate.

---

## Core Principle: Derive, Don't Hardcode

**This skill must never be a source of truth for anything the pipeline already defines.** Field names, thresholds, file names, expected ranges, and schemas all change as the pipeline evolves. The skill discovers these at runtime from two sources:

1. **Pipeline source code** (schemas, configuration, stage contracts)
2. **Output data** (actual file contents, record structures, `_meta` blocks)

Every evaluation phase starts with discovery. Python snippets in this skill express **intent** ("print all triples in subject → predicate → object form"), not **field names** ("access `t['subject_label']`"). When this skill says "the field for X," it means "find the field that represents X by inspecting the data."

---

## Finding Analysis Protocol

**Every finding MUST receive a full deep-dive analysis.** Do not shortcut to a label and severity. For each finding, work through this structure before moving to the next:

1. **Symptom** — What did you observe in the output? Quote specific data (text, counts, field values). Be precise enough that someone unfamiliar with the run could see it.

2. **Issue** — What is wrong? Distinguish between the observed symptom and the underlying problem. "Entity count is high" is a symptom. "Entity resolution is not merging variants because the similarity threshold is too high for the current embedding model" is the issue.

3. **Root Cause** — Where in the pipeline does this originate? Trace backward through the stage chain. A problem visible in Stage 9 output may originate in Stage B. Name the specific stage, and if possible the specific function or logic path. Read the relevant pipeline source code to confirm — do not guess.

4. **Cascade Analysis** — What downstream effects does this root cause produce? Trace forward through the stage chain. A Stage B entity type error cascades to Stage C (wrong Faker generator), Stage 5 (wrong entity type in resolution), Stage 6 (garbled triple subjects), and Stage 9 (nonsensical graph labels). Map the full chain.

5. **Architectural Fix** — What is the right fix? Not a patch — a fix that solves the problem permanently. Consider:
   - Does the fix address the root cause, not just the symptom?
   - Will the fix survive future pipeline changes (new stages, new models, new schemas)?
   - Does the fix minimize technical debt, or does it add a special case that someone will need to maintain?
   - Are there simpler alternatives that achieve the same outcome?
   - What are the risks of the fix itself (could it break something else)?

6. **Verification** — How would you confirm the fix worked? What specific output would change, and what would the correct values look like?

**Do not skip this protocol.** A finding without root cause analysis is just noise. A finding without cascade analysis misses the real impact. A finding without an architectural fix creates technical debt.

---

## Evaluation Process

Execute ALL phases in order. Use Bash for JSON parsing with python3 where needed. Keep individual Bash calls focused and fast.

---

### Phase 0: Discovery & Orientation

This phase builds the evaluation context that all subsequent phases depend on. Do not skip any step.

#### 0A: Locate Pipeline Source Code

1. **Find the pipeline root.** The output directory is typically under a project that contains a `pipeline/` directory. Look for `pipeline/config.yaml` relative to the output directory, or in common locations:
   - Same repo as the output directory
   - `C:\Users\Troy Davis\dev\contact-center-lab\pipeline`
   - Ask the user if not found.

2. **Read `pipeline/config.yaml`** in full. Extract and note:
   - LLM backend configuration (which model, concurrency, timeout)
   - Embedding backend configuration (model name, dimension, truncate_dimension)
   - All pipeline thresholds (`semantic_break_threshold`, `typing_min_confidence`, `decomposition_min_length`, `dedup_high_threshold`, `dedup_review_threshold`, `entity_similarity_threshold`, `predicate_similarity_threshold`, etc.)
   - Sanitization config: denylist terms, safelist references, internal URL domains
   - Predicate synonyms and blocklist

3. **Read `pipeline/shared/models.py`**. Identify all dataclass definitions and their field names. These are the canonical schemas for every stage output. Note especially:
   - Triple fields (the field names for subject, predicate, object, condition, negation)
   - NormalizedProcedure fields (the field names for title/name, goal/purpose/expected_outcome, steps)
   - FinalAtom fields (especially `procedure`, `triples`, `entity_refs`)
   - EntityRecord fields
   - PolicyRecord and FormalRule fields

4. **Read `pipeline/runner.py`** to discover:
   - `STAGE_ORDER` — what stages exist and their execution order
   - `STAGE_NAMES` — human-readable stage names
   - `CONCURRENT_GROUPS` — which stages run in parallel
   - Output file naming patterns per stage

#### 0B: Scan Output Directory

1. **List all files** in the output directory (including `final/` subdirectory). Record which stage output files are present. Note any unexpected or missing files.

2. **For each stage output file found**, read the `_meta` record (typically the last line in JSONL files, or a `_meta` key in JSON files). Record:
   - Stage number/letter
   - Timestamp
   - All counts and rates (these are the pipeline's own self-reported metrics)
   - LLM failure counts and rates (for LLM-calling stages)
   - Any error or failure flags (e.g., `hdbscan_failed`)

3. **Read `final/statistics.json`** in full. This is the pipeline's summary dashboard. Record its complete structure — do not assume key names; inspect the actual JSON.

4. **Discover schemas from actual data.** For each key output file, read the first non-`_meta` record and note its field names. This is the ground truth for how to access data in subsequent phases, regardless of what `models.py` says the fields should be called.

#### 0C: Compute Expectations

1. **Identify article count** from statistics.json (look for a key representing document/article count).

2. **Derive expected ranges** using these per-article heuristics for well-formed KB articles. These are evaluation heuristics, not pipeline config — they represen