eval-rag

What this skill does

# Eval RAG

Evaluate RAG pipelines by separating retrieval quality from generation quality. Fix retrieval first.

## Entry Point

When this skill is invoked, start with:

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 EVAL RAG
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Retrieval and generation fail differently. Measure them separately.

What RAG feature are we evaluating?
What does it retrieve from? (knowledge base, docs, database)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

## Prerequisites

Complete error analysis (`/upgrade-evals`) on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first — the LLM can ignore irrelevant context but cannot generate from missing context.

## Core Process

### Step 1: Separate Retrieval from Generation

Ask the PM: "When the AI gives a wrong answer, is it because it found the wrong documents, or because it had the right documents but said the wrong thing?"

This determines where to focus:

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| AI says "I don't know" or gives generic answer | Retrieval: relevant docs not found | Improve chunking, embeddings, or query |
| AI confidently states something not in any doc | Generation: hallucination | Improve prompt, add grounding |
| AI answers a different question than asked | Could be either | Check retrieved docs first |

### Step 2: Build a Retrieval Evaluation Dataset

Queries paired with ground-truth relevant document chunks.

**Manual curation (highest quality, PM-driven):**
Ask the PM: "For each of these test queries, which document(s) actually contain the answer?"

The PM maps queries to chunks. This is domain knowledge work — the PM knows the content better than any automated system.

**Synthetic QA generation (scalable):**

Claude Code executes: For each document chunk, extract a fact and generate a question answerable only from that fact.

```
Given a chunk of text, extract a specific, self-contained fact from it.
Then write a question that is directly and unambiguously answered
by that fact alone.

Return output in JSON format:
{ "fact": "...", "question": "..." }

Chunk: "{text_chunk}"
```

**Adversarial questions (stress-testing):**

Claude Code executes: Create queries that resemble content in multiple chunks but are only answered by one. Find similar chunks via embedding search, then generate a question that only the target chunk answers.

### Step 3: Measure Retrieval Quality

Claude Code executes all metric computations. The PM interprets results.

**Recall@k** — Did we find the right documents?
```
Recall@k = (relevant docs in top k) / (total relevant docs for query)
```
Prioritize for first-pass retrieval. High recall = the right docs are in the candidate set.

**Precision@k** — How much noise in the results?
```
Precision@k = (relevant docs in top k) / k
```
Prioritize for reranking evaluation.

**MRR (Mean Reciprocal Rank)** — How quickly do we find the first right document?
```
MRR = (1/N) * sum(1/rank_of_first_relevant_doc)
```
Best for single-fact lookups.

**NDCG@k** — Are the most relevant results ranked highest?
```
DCG@k  = sum over i=1..k of: rel_i / log2(i+1)
IDCG@k = DCG@k with documents sorted by decreasing relevance
NDCG@k = DCG@k / IDCG@k
```
Use when documents have varying utility.

**Which metric to use:**

| Your Query Type | Primary Metric | Why |
|----------------|---------------|-----|
| Single-fact lookups ("What's our refund policy?") | MRR | One key chunk needed, want it ranked first |
| Broad coverage ("Summarize market trends") | Recall@k | Need all relevant docs, even at cost of noise |
| Ranked results matter | NDCG@k or Precision@k | Quality of ranking matters for the answer |

**Choosing k:** Factual lookup uses k=1-2. Synthesis query uses k=5-10. Ask the PM: "How many documents does your pipeline pass to the LLM?"

### Step 4: Optimize Chunking (if retrieval is the bottleneck)

Treat chunking as a tunable hyperparameter. Claude Code executes a grid search.

Claude Code executes:

```
Test combinations of chunk size and overlap.
Re-index the corpus for each configuration.
Measure retrieval metrics on the evaluation dataset.
```

Example results:

| Chunk Size | Overlap | Recall@5 | NDCG@5 |
|-----------|---------|----------|--------|
| 128 tokens | 0 | 0.82 | 0.69 |
| 128 tokens | 64 | 0.88 | 0.75 |
| 256 tokens | 0 | 0.86 | 0.74 |
| 256 tokens | 128 | 0.89 | 0.77 |
| 512 tokens | 0 | 0.80 | 0.72 |

Present to PM: "256 tokens with 128 overlap gives the best recall. Want to go with that?"

**Content-aware chunking:** When fixed-size chunks split related information, use natural document boundaries (sections, paragraphs). Augment chunks with context: prepend document title and section headings before embedding.

### Step 5: Evaluate Generation Quality

After confirming retrieval works, evaluate what the LLM does with the retrieved context.

**Answer faithfulness** — Does the output accurately reflect the retrieved context?
- **Hallucinations:** Information absent from source documents. In RAG, even correct facts from the LLM's own knowledge count as hallucinations.
- **Omissions:** Relevant information from the context ignored in the output.
- **Misinterpretations:** Context information represented inaccurately.

**Answer relevance** — Does the output address the original query? An answer can be faithful to the context but fail to answer what the user asked.

Use `/upgrade-evals` to discover specific manifestations in your pipeline. Use `/build-judge` for failure modes that require subjective evaluation.

### Step 6: Diagnose with the Metric Pattern Table

Present this to the PM for root cause identification:

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 RAG DIAGNOSTIC TABLE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

| Context Quality | Faithfulness | Relevance | Diagnosis |
|-----------------|-------------|-----------|-----------|
| Good            | Good        | Bad       | Generator attended to wrong section |
| Good            | Bad         | --        | Hallucination or misinterpretation |
| Bad             | --          | --        | Retrieval problem. Fix first. |

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

**What to tell your engineer:**
- "Context quality is low" → Fix chunking, embeddings, or query preprocessing.
- "Faithful but not relevant" → Generator needs better instruction to address the actual question.
- "Not faithful" → Add grounding instructions, reduce temperature, or use citation enforcement.

## Multi-Hop Retrieval

For queries requiring information from multiple chunks:

**Two-hop Recall@k:** Fraction of 2-hop queries where both ground-truth chunks appear in top k results.

```
TwoHopRecall@k = (1/N) * sum(1 if {Chunk1, Chunk2} ⊆ top_k_results)
```

Diagnose: classify failures as hop 1 miss, hop 2 miss, or rank-out-of-top-k.

## Output

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 RAG EVALUATION RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Feature: [name]
Eval dataset: [N] queries with ground-truth mappings

RETRIEVAL:
| Metric | Score | Interpretation |
|--------|-------|---------------|
| Recall@5 | [X] | [good/needs work] |
| Precision@5 | [X] | [good/needs work] |
| MRR | [X] | [good/needs work] |

GENERATION:
| Dimension | Pass Rate | Top Failure Mode |
|-----------|-----------|-----------------|
| Faithfulness | [%] | [description] |
| Relevance | [%] | [description] |

DIAGNOSIS: [Retrieval problem / Generation problem / Both]

RECOMMENDED ACTIONS:
1. [highest-impact fix]
2. [second fix]
3. [third fix]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

## Anti-Patterns

- Using a single end-to-end correctness metric without separating retrieval and g