writing-evals

What this skill does

# Writing Evals

You write evaluations that prove AI capabilities work. Evals are the test suite for non-deterministic systems: they measure whether a capability still behaves correctly after every change.

## Prerequisites

- Complete the [Axiom AI SDK Quickstart](https://axiom.co/docs/ai-engineering/quickstart) (instrumentation + authentication)

Verify the SDK is installed:

```bash
ls node_modules/axiom/dist/
```

If not installed, install it using the project's package manager (e.g., `pnpm add axiom`).

**Always check `node_modules/axiom/dist/docs/` first** for the correct API signatures, import paths, and patterns for the installed SDK version. The bundled docs are the source of truth — do not rely on the examples in this skill if they conflict.

## Philosophy

1. **Evals are tests for AI.** Every eval answers: "does this capability still work?"
2. **Scorers are assertions.** Each scorer checks one property of the output.
3. **Flags are variables.** Flag schemas let you sweep models, temperatures, strategies without code changes.
4. **Data drives coverage.** Happy path, adversarial, boundary, and negative cases.
5. **Validate before running.** Never guess import paths or types—use reference docs.

---

## Axiom Terminology

| Term | Definition |
|------|------------|
| **Capability** | A generative AI system that uses LLMs to perform a specific task. Ranges from single-turn model interactions → workflows → single-agent → multi-agent systems. |
| **Collection** | A curated set of reference records used for testing and evaluation of a capability. The `data` array in an eval file is a collection. |
| **Collection Record** | An individual input-output pair within a collection: `{ input, expected, metadata? }`. |
| **Ground Truth** | The validated, expert-approved correct output for a given input. The `expected` field in a collection record. |
| **Scorer** | A function that evaluates a capability's output, returning a score. Two types: **reference-based** (compares output to expected ground truth) and **reference-free** (evaluates quality without expected values, e.g., toxicity, coherence). |
| **Eval** | The process of testing a capability against a collection using scorers. Three modes: **offline** (against curated test cases), **online** (against live production traffic), **backtesting** (against historical production traces). |
| **Flag** | A configuration parameter (model, temperature, strategy) that controls capability behavior without code changes. |
| **Experiment** | An evaluation run with a specific set of flag values. Compare experiments to find optimal configurations. |

---

## How to Start

When the user asks you to write evals for an AI feature, **read the code first**. Do not ask questions — inspect the codebase and infer everything you can.

### Step 1: Understand the feature

1. **Find the AI function** — search for the function the user mentioned. Read it fully.
2. **Trace the inputs** — what data goes in? A string prompt, structured object, conversation history?
3. **Trace the outputs** — what comes back? A string, category label, structured object, agent result with tool calls?
4. **Identify the model call** — which LLM/model is used? What parameters (temperature, maxTokens)?
5. **Check for existing evals** — search for `*.eval.ts` files. Don't duplicate what exists.
6. **Check for app-scope** — look for `createAppScope`, `flagSchema`, `axiom.config.ts`.

### Step 2: Determine eval type

Based on what you found:

| Output type | Eval type | Scorer pattern |
|-------------|-----------|----------------|
| String category/label | Classification | Exact match |
| Free-form text | Text quality | Contains keywords or LLM-as-judge |
| Array of items | Retrieval | Set match |
| Structured object | Structured output | Field-by-field match |
| Agent result with tool calls | Tool use | Tool name presence |
| Streaming text | Streaming | Exact match or contains (auto-concatenated) |

### Step 3: Choose scorers

Every eval needs **at least 2 scorers**. Use this layering:

1. **Correctness scorer (required)** — Does the output match expected? Pick from the eval type table above (exact match, set match, field match, etc.).
2. **Quality scorer (recommended)** — Is the output well-formed? Check confidence thresholds, output length, format validity, or field completeness.
3. **Reference-free scorer (add for user-facing text)** — Is the output coherent, relevant, non-toxic? Use LLM-as-judge or autoevals.

| Output type | Minimum scorers |
|-------------|----------------|
| Category label | Correctness (exact match) + Confidence threshold |
| Free-form text | Correctness (contains/Levenshtein) + Coherence (LLM-as-judge) |
| Structured object | Field match + Field completeness |
| Tool calls | Tool name presence + Argument validation |
| Retrieval results | Set match + Relevance (LLM-as-judge) |

### Step 4: Generate

1. Create the `.eval.ts` file colocated next to the source file
2. Import the actual function — do not create a stub
3. Write the scorers based on the output type (minimum 2, see step 3)
4. Generate test data (see Data Design Guidelines)
5. Set capability and step names matching the feature's purpose
6. If flags exist, use `pickFlags` to scope them

### Only ask if you cannot determine:
- What "correct" means for ambiguous outputs (e.g., summarization quality)
- Whether the user wants pass/fail or partial credit scoring
- Which parameters should be tunable via flags (if not already using flags)

---

## Project Layout

### Recommended: Colocated with source

Place `.eval.ts` files next to their implementation files, organized by capability:

```
src/
├── lib/
│   ├── app-scope.ts
│   └── capabilities/
│       └── support-agent/
│           ├── support-agent.ts
│           ├── support-agent-e2e-tool-use.eval.ts
│           ├── categorize-messages.ts
│           ├── categorize-messages.eval.ts
│           ├── extract-ticket-info.ts
│           └── extract-ticket-info.eval.ts
axiom.config.ts
package.json
```

### Minimal: Flat structure

For small projects, keep everything in `src/`:

```
src/
├── app-scope.ts
├── my-feature.ts
└── my-feature.eval.ts
axiom.config.ts
package.json
```

The default glob `**/*.eval.{ts,js}` discovers eval files anywhere in the project. `axiom.config.ts` always lives at the project root.

---

## Eval File Structure

Standard structure of an eval file:

```typescript
import { pickFlags } from '@/app-scope';       // or relative path
import { Eval } from 'axiom/ai/evals';
import { Scorer } from 'axiom/ai/scorers';
import { Mean, PassHatK } from 'axiom/ai/scorers/aggregations';
import { myFunction } from './my-function';

const MyScorer = Scorer('my-scorer', ({ output, expected }: { output: string; expected: string }) => {
  return output === expected;
});

Eval('my-eval-name', {
  capability: 'my-capability',
  step: 'my-step',                              // optional
  configFlags: pickFlags('myCapability'),        // optional, scopes flag access
  data: [
    { input: '...', expected: '...', metadata: { purpose: '...' } },
  ],
  task: async ({ input }) => {
    return await myFunction(input);
  },
  scorers: [MyScorer],
});
```

---

## Reference

For detailed patterns and type signatures, read these on demand:

- `reference/scorer-patterns.md` — All scorer patterns (exact match, set match, structured, tool use, autoevals, LLM-as-judge), score return types, typing tips
- `reference/api-reference.md` — Full type signatures, import paths, aggregations, streaming tasks, dynamic data loading, manual token tracking, CLI options
- `reference/flag-schema-guide.md` — Flag schema rules, validation, `pickFlags`, CLI overrides, common patterns
- `reference/templates/` — Ready-to-use eval file templates (see Templates section below)

---

## Authentication Setup

Before running evals, the user must authenticate. Check if they've already done this before suggesting it.

Set environment variables (works for both offlin