output-eval-validate-judge

What this skill does

# Validating LLM Judges

## Overview

An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this **before** trusting any `judgeVerdict()`, `judgeScore()`, or `judgeLabel()` evaluator in your eval suite.

## Prerequisites

1. **A judge `.prompt` file** — Written following `output-eval-judge-prompt`
2. **~100 human-labeled traces** — With binary pass/fail labels for the failure mode this judge targets. Aim for ~50 pass and ~50 fail. Minimum: 20 pass and 20 fail.
3. **Labels stored in dataset YAML** — Each dataset has `ground_truth.evals.<evaluator_name>.verdict: pass` or `fail`

This process applies **only to LLM-based judges**. For code-based `Verdict.*` evaluators, write unit tests instead.

## Step 1: Create Data Splits

Split your labeled datasets into three groups:

| Split | % of Data | Purpose | Example (100 datasets) |
|-------|-----------|---------|----------------------|
| **Train** | 10-20% | Source of few-shot examples in the judge prompt | 15 datasets |
| **Dev** | 40-45% | Iterate on judge prompt, measure TPR/TNR | 42 datasets |
| **Test** | 40-45% | Final held-out measurement, run once | 43 datasets |

### Organizing splits

Use a naming convention or subdirectories to separate splits:

**Option A: Name prefixes**
```
tests/datasets/
├── train_formal_pass_01.yml
├── train_casual_fail_01.yml
├── dev_technical_pass_01.yml
├── dev_ambiguous_fail_01.yml
├── test_simple_pass_01.yml
├── test_contradictory_fail_01.yml
└── ...
```

**Option B: Subdirectories**
```
tests/datasets/
├── train/
│   ├── formal_pass_01.yml
│   └── casual_fail_01.yml
├── dev/
│   ├── technical_pass_01.yml
│   └── ambiguous_fail_01.yml
└── test/
    ├── simple_pass_01.yml
    └── contradictory_fail_01.yml
```

### Splitting rules

- **Balance pass/fail in each split** — Don't put all failures in dev and all passes in test
- **Randomize** — Don't sort by difficulty or topic
- **Training examples in the prompt** — Use only train-split examples as few-shot in the judge `.prompt` file. Never use dev or test examples — that's data leakage
- **Lock the test split** — Once created, do not look at test data until final measurement

## Step 2: Run the Judge on Dev Set

Execute the eval workflow against only the dev-split datasets:

```bash
# Run with cached output on dev datasets
npx output workflow test <workflowName> --cached \
  --dataset dev_technical_pass_01,dev_ambiguous_fail_01,dev_formal_pass_02,...
```

Or if using subdirectories, list the dev dataset names:

```bash
npx output workflow test <workflowName> --cached \
  --dataset $(ls tests/datasets/dev/ | sed 's/.yml//' | tr '\n' ',')
```

Save the output. You need the judge's verdict for each dataset to compare against ground truth.

### Extracting results

Use `--format json` to get machine-readable results:

```bash
npx output workflow test <workflowName> --cached --dataset <dev_datasets> --format json
```

The output includes per-dataset, per-evaluator verdicts that you can compare against `ground_truth.evals.<evaluator_name>.verdict`.

## Step 3: Compute TPR and TNR

For the evaluator you're validating, build a confusion matrix from the dev results.

### Definitions

Using "fail" as the positive class (what you're trying to detect):

| | Judge says Fail | Judge says Pass |
|---|---|---|
| **Human says Fail** | True Positive (TP) | False Negative (FN) |
| **Human says Pass** | False Positive (FP) | True Negative (TN) |

**TPR (True Positive Rate)** = TP / (TP + FN)
- "Of all the real failures, what fraction did the judge catch?"
- Low TPR means the judge **misses real failures** (dangerous)

**TNR (True Negative Rate)** = TN / (TN + FP)
- "Of all the real passes, what fraction did the judge correctly approve?"
- Low TNR means the judge **flags passing traces as failures** (noisy)

### Example computation

Dev set results for `check_tone` evaluator (42 datasets):

| | Judge: Fail | Judge: Pass |
|---|---|---|
| **Human: Fail** | 18 (TP) | 3 (FN) |
| **Human: Pass** | 2 (FP) | 19 (TN) |

- TPR = 18 / (18 + 3) = **85.7%**
- TNR = 19 / (19 + 2) = **90.5%**

### Why not raw accuracy?

Raw accuracy = (TP + TN) / total = (18 + 19) / 42 = 88.1%

This looks fine, but masks problems. If your dataset were 90% pass (class imbalance), a judge that always says "pass" would get 90% accuracy while catching zero failures (TPR = 0%). TPR and TNR measure what actually matters: catching failures and not crying wolf.

## Step 4: Inspect Disagreements

For every case where the judge disagrees with the human label, determine the root cause.

### False Negatives (judge missed a real failure)

The judge said "pass" but the human said "fail." For each:

1. Read the trace and the judge's critique
2. Determine why the judge missed it:
   - **Criterion too narrow** — The prompt defines failure too narrowly. Broaden the fail definition.
   - **Missing few-shot example** — The failure pattern isn't represented in examples. Add a similar borderline example from the train split.
   - **Insufficient context** — The judge doesn't have the information needed to detect this failure. Add the missing variable to the prompt.

### False Positives (judge flagged a passing trace)

The judge said "fail" but the human said "pass." For each:

1. Read the trace and the judge's critique
2. Determine why the judge flagged it:
   - **Criterion too broad** — The prompt defines failure too broadly. Tighten the fail definition.
   - **Misleading few-shot example** — A borderline example is being overgeneralized. Clarify or replace it.
   - **Overly strict** — The judge applies the criterion more strictly than intended. Add explicit exceptions to the prompt.

### Logging disagreements

Track each disagreement to guide prompt iteration:

| Dataset | Human | Judge | Root Cause | Fix |
|---------|-------|-------|------------|-----|
| dev_technical_pass_03 | pass | fail | Judge flagged "it's" as casual but context was a direct quote | Add exception: "Contractions within direct quotes are acceptable" |
| dev_ambiguous_fail_02 | fail | pass | Judge missed subtle tone shift in paragraph 3 | Add borderline few-shot example showing mid-text tone drift |

## Step 5: Iterate

Apply the fixes from Step 4 to the judge `.prompt` file. Then re-run on the dev set:

```bash
npx output workflow test <workflowName> --cached --dataset <dev_datasets>
```

Recompute TPR and TNR. Repeat until both metrics meet the target.

### Targets

| Metric | Target | Minimum Acceptable |
|--------|--------|--------------------|
| TPR | > 90% | > 80% |
| TNR | > 90% | > 80% |

If you can't reach 80%/80% after 3-4 iterations:

1. **Upgrade the model** — Switch from Haiku to Sonnet in the `.prompt` frontmatter
2. **Split the criterion** — The failure mode may contain two distinct sub-failures that need separate judges
3. **Revisit the labels** — Some human labels may be inconsistent. Re-label disagreements with a second reviewer

### Iteration checklist

Each iteration:
- [ ] Identified root cause for each disagreement
- [ ] Applied targeted fix to `.prompt` file (not random changes)
- [ ] Re-ran on dev set
- [ ] Recomputed TPR and TNR
- [ ] Logged the iteration number, changes made, and resulting metrics

## Step 6: Final Measurement on Test Set

Once dev metrics meet the target, run the judge on the held-out test set **exactly once**:

```bash
npx output workflow test <workflowName> --cached --dataset <test_datasets> --format json
```

Compute TPR and TNR on the test results. Record these as the final metrics.

### Interpreting final results

- **Test metrics close to dev metrics** — The judge generalizes well. Ship it.
- **Test metrics significantly lower** — The judge may be overfit to dev set patterns. Do not iterate on the test set. Instead, gather more labeled data, re-split, and restart from Step 2.

### Recording results

Document the final