output-eval-dataset-design

What this skill does

# Designing Eval Datasets

## Overview

Diverse datasets catch more failures. This skill teaches dimension-based dataset design — systematically varying inputs along axes that target failure-prone regions of your workflow. The output is a set of YAML dataset files ready for `output workflow test`.

## When to Use

- Bootstrapping an eval dataset for a new workflow
- Existing datasets only cover happy paths
- Real production traces are sparse (fewer than 50)
- Stress-testing specific failure hypotheses from error analysis

## When NOT to Use

- You already have 100+ representative real traces — use stratified sampling from those instead of generating synthetic data
- You haven't done error analysis yet — do that first (`output-eval-error-analysis`) so your dimensions target real failure modes, not guesses

## Step 1: Define Dimensions

Identify 3+ axes of input variation that target **anticipated failure modes**. Each dimension should vary one aspect of the input that you expect to influence output quality.

### Deriving dimensions from error analysis

Map failure categories to input properties that trigger them:

| Failure Category | Triggering Input Property | Dimension |
|-----------------|---------------------------|-----------|
| Off-topic drift | Ambiguous or broad topics | Topic specificity: specific / broad / ambiguous |
| Tone mismatch | Conflicting tone signals | Tone difficulty: simple / nuanced / contradictory |
| Too short | Short or vague input | Input detail: minimal / moderate / comprehensive |
| Missing sections | Many explicit requirements | Requirement count: 0 / 1-2 / 5+ |
| Hallucinated URLs | Technical topics with real entities | Entity density: none / few / many |

### Example dimensions for a blog generation workflow

| Dimension | Values | Why |
|-----------|--------|-----|
| Topic complexity | simple, technical, ambiguous | Technical and ambiguous topics trigger more hallucination and drift |
| Tone request | none, formal, casual, contradictory | Explicit tone requests reveal tone-matching failures |
| Length constraint | none, short (100w), long (1000w) | Extreme length constraints trigger truncation and padding |
| Required sections | none, 1 section, 3+ sections | Multiple required sections stress structural compliance |

Aim for 3-5 dimensions. More than 5 creates an unmanageable combinatorial space.

## Step 2: Draft Tuples

Create ~20 combinations of dimension values. Cover the extremes and the combinations most likely to cause failures.

### Tuple selection strategy

1. **Cover every dimension value at least twice** — ensures no blind spots
2. **Pair difficult values together** — "ambiguous topic + contradictory tone + 3+ required sections" is where failures cluster
3. **Include a few easy combinations** — confirms the workflow works under normal conditions
4. **Avoid near-duplicates** — each tuple should test a distinct scenario

### Example tuples

| # | Topic Complexity | Tone | Length | Required Sections |
|---|-----------------|------|--------|-------------------|
| 1 | simple | none | none | none |
| 2 | simple | formal | short | 1 section |
| 3 | technical | formal | long | 3+ sections |
| 4 | technical | casual | short | none |
| 5 | ambiguous | formal | long | 1 section |
| 6 | ambiguous | contradictory | none | 3+ sections |
| 7 | simple | contradictory | long | none |
| 8 | technical | none | none | 3+ sections |
| 9 | ambiguous | casual | short | 1 section |
| 10 | simple | casual | long | 3+ sections |
| 11 | technical | formal | short | 1 section |
| 12 | ambiguous | none | long | none |
| 13 | simple | formal | none | 3+ sections |
| 14 | technical | contradictory | long | 1 section |
| 15 | ambiguous | formal | short | 3+ sections |
| 16 | simple | none | short | 1 section |
| 17 | technical | casual | long | 3+ sections |
| 18 | ambiguous | contradictory | short | none |
| 19 | technical | formal | none | none |
| 20 | ambiguous | casual | long | 3+ sections |

Review each tuple and ask: "Is this a realistic scenario a user might create?" Discard any that aren't.

## Step 3: Convert Tuples to Workflow Inputs

Transform each tuple into a JSON object matching the workflow's `inputSchema` from `types.ts`.

### Read the schema first

```bash
# Find the input schema
cat src/workflows/<workflow_name>/types.ts
```

### Manual conversion (small datasets)

For each tuple, write the corresponding JSON input:

**Tuple 3:** technical + formal + long + 3+ sections

```json
{
  "topic": "Quantum error correction techniques in superconducting qubit architectures",
  "tone": "formal",
  "min_length": 1000,
  "required_sections": ["Introduction", "Technical Background", "Current Approaches", "Challenges", "Conclusion"]
}
```

**Tuple 6:** ambiguous + contradictory + none + 3+ sections

```json
{
  "topic": "Things that matter",
  "tone": "Write in a formal academic style but keep it super casual and fun",
  "required_sections": ["Overview", "Deep Dive", "Takeaways"]
}
```

Use realistic, natural-sounding inputs. Avoid test-looking data like "Test topic 1" or "Lorem ipsum."

### LLM-assisted conversion (larger datasets)

For 20+ tuples, use an LLM to batch-convert tuples into realistic inputs. Create a prompt that takes the tuple values and the `inputSchema`, then generates a natural JSON input. Review every generated input for realism before using it.

## Step 4: Generate Dataset Files

Run each input through the workflow to capture real output:

```bash
# Generate datasets one at a time
npx output workflow dataset generate blog_generator \
  --input '{"topic": "Quantum error correction", "tone": "formal", "min_length": 1000}' \
  --name technical_formal_long

npx output workflow dataset generate blog_generator \
  --input '{"topic": "Things that matter", "tone": "Write formally but keep it casual"}' \
  --name ambiguous_contradictory
```

Each command creates a YAML file in `tests/datasets/` with `input` and `last_output` populated.

### Naming convention

Use names that encode the key dimensions:

```
tests/datasets/
├── simple_no_constraints.yml
├── simple_formal_short.yml
├── technical_formal_long_sections.yml
├── technical_casual_short.yml
├── ambiguous_formal_long.yml
├── ambiguous_contradictory_sections.yml
└── ...
```

### Batch generation

For many datasets, create a shell script:

```bash
#!/bin/bash
# generate_datasets.sh

inputs=(
  '{"topic": "Solar panels", "tone": "formal"}|simple_formal'
  '{"topic": "Quantum error correction in superconducting qubits", "tone": "casual", "min_length": 1000}|technical_casual_long'
  '{"topic": "Things that matter", "required_sections": ["Overview", "Dive", "Takeaways"]}|ambiguous_sections'
)

for entry in "${inputs[@]}"; do
  IFS='|' read -r input name <<< "$entry"
  echo "Generating: $name"
  npx output workflow dataset generate blog_generator --input "$input" --name "$name"
done
```

## Step 5: Add Ground Truth

After generation, edit each dataset YAML to add `ground_truth` labels. Review the `last_output` and assign verdicts per evaluator.

```yaml
name: ambiguous_contradictory_sections
input:
  topic: "Things that matter"
  tone: "Write in a formal academic style but keep it super casual and fun"
  required_sections: ["Overview", "Deep Dive", "Takeaways"]
last_output:
  output:
    title: "Things That Matter: A Casual Academic Exploration"
    blog_post: "So here's the thing about stuff that matters..."
  executionTimeMs: 4200
  date: '2026-03-25T00:00:00.000Z'
ground_truth:
  human_verdict: fail
  notes: "Contradictory tone caused drift; missing 'Deep Dive' section"
  evals:
    check_tone:
      expected_tone: "formal academic style"
      verdict: fail
    check_topic:
      required_topic: "Things that matter"
      verdict: partial
    check_sections:
      required_sections: ["Overview", "Deep Dive", "Takeaways"]
      verdict: fail
    check_length:
      min_length: 100
      verdict: pass
```

### Labeling tips

- Label the **global `human_verdict`** for