generate-test-data

What this skill does

# Generate Test Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline. Dimension-based tuples, not random generation.

## Entry Point

When this skill is invoked, start with:

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 GENERATE TEST DATA
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Diverse inputs expose the failure space. Random generation doesn't.

What AI feature are we generating test data for?
What kinds of inputs does it take?

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

## Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the PM about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

## Core Process

### Step 1: Define Dimensions

Dimensions are axes of variation specific to the application. The PM defines these — they know where failures happen.

```
Dimension 1: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name] — [What it captures]
  Values: [value_a, value_b, value_c, ...]
```

Example for a customer support chatbot:

```
Query Type: what the user is asking about
  Values: [billing, technical issue, account access, feature request, cancellation]

User Expertise: how technical the user is
  Values: [non-technical, somewhat technical, power user]

Complexity: how many steps to resolve
  Values: [single-step, multi-step, requires escalation]
```

Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.

Ask the PM: "What are the 3 most important ways inputs vary for your feature? Think about what makes some inputs harder than others."

### Step 2: Draft 20 Tuples with the PM

A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the PM and iterate until they confirm the tuples reflect realistic scenarios.

```
(Query Type: Billing, User Expertise: Non-technical, Complexity: Multi-step)
(Query Type: Technical Issue, User Expertise: Power User, Complexity: Single-step)
(Query Type: Cancellation, User Expertise: Non-technical, Complexity: Requires Escalation)
```

The PM's domain knowledge is essential. They know which combinations actually occur and which are unrealistic.

Claude Code executes: Generate the initial 20 tuples ensuring coverage across dimension values. Present to PM for validation.

### Step 3: Expand Tuples with LLM

Claude Code executes: Generate additional tuples using the PM-validated set as examples.

```
Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.
```

### Step 4: Convert Tuples to Natural Language Queries

Separate step from tuple generation. Single-step generation (tuples + queries together) produces repetitive phrasing.

Claude Code executes: Convert each tuple to a realistic user query using a separate prompt per tuple.

```
We are generating synthetic user queries for a {application}.
{Brief description of what it does.}

Given:
{dim1}: {value}
{dim2}: {value}
{dim3}: {value}

Write a realistic query that a user might enter. The query should
reflect the specified characteristics.

Example: "{one of the PM-written examples}"

Now generate a new query.
```

### Step 5: Filter for Quality

Review generated queries with the PM. Discard and regenerate when:
- Phrasing is awkward or unrealistic.
- Content doesn't match the tuple's intent.
- Queries are too similar to each other.

Claude Code executes: Rate realism using an LLM, discard below threshold, regenerate replacements.

### Step 6: Run Through Pipeline

Execute all queries through the full LLM pipeline. Capture complete traces: input, all intermediate steps, tool calls, retrieved docs, final output.

**Target: ~100 high-quality, diverse traces.** This is a rough heuristic for reaching saturation.

Claude Code executes: Run the queries, capture traces, format for analysis. These traces feed directly into `/upgrade-evals` for error analysis.

## Output

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 TEST DATA GENERATED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Feature: [name]
Dimensions: [dim1], [dim2], [dim3]
Tuples generated: [count]
Queries generated: [count]
Queries after filtering: [count]

DIMENSION COVERAGE:
| Dimension | Values Covered | Gaps |
|-----------|---------------|------|
| [dim1]    | [X/Y]        | [any missing] |
| [dim2]    | [X/Y]        | [any missing] |
| [dim3]    | [X/Y]        | [any missing] |

NEXT STEPS:
- Run /upgrade-evals on these traces for error analysis
- Run /build-judge for failure modes that need automated evaluation

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

## When Real Data Exists

When you have real queries available, don't just sample randomly. Use stratified sampling:

1. **Identify high-variance dimensions** — read through queries and find ways they differ (length, topic, complexity, presence of constraints).
2. **Assign labels** — for small sets, with the PM; for large sets, use K-means clustering on query embeddings.
3. **Sample from each group** — ensures coverage across query types, not just the most common ones.

Use synthetic data to fill gaps in underrepresented query types.

## Anti-Patterns

- Unstructured generation. "Give me test queries" without dimensions produces generic, repetitive, happy-path examples.
- Single-step generation. Generating tuples and queries in one prompt produces less diverse results.
- Arbitrary dimensions. Dimensions that don't target failure-prone regions waste test budget.
- Skipping PM review of tuples. Without the PM validating tuples, you can't judge realism.
- Synthetic data when no one can judge realism. If no one can tell whether a synthetic trace is realistic, use real data.
- Synthetic data for complex domain-specific content (legal filings, medical records) where LLMs miss structural nuance.

---

**Methodology:** Adapted from Hamel Husain's generate-synthetic-data skill (evals-skills, MIT license)
**PM adaptation:** PM defines dimensions and validates realism, Claude Code handles generation and pipeline execution