rag-architecture

What this skill does

# RAG Architecture

## When to Use This Skill

Use this skill when:

- Designing RAG pipelines for LLM applications
- Choosing chunking and embedding strategies
- Optimizing retrieval quality and relevance
- Building knowledge-grounded AI systems
- Implementing hybrid search (dense + sparse)
- Designing multi-stage retrieval pipelines

**Keywords:** RAG, retrieval-augmented generation, embeddings, chunking, vector search, semantic search, context window, grounding, knowledge base, hybrid search, reranking, BM25, dense retrieval

## RAG Architecture Overview

```text
┌─────────────────────────────────────────────────────────────────────┐
│                       RAG Pipeline                                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐  │
│  │   Ingestion  │    │   Indexing   │    │    Vector Store      │  │
│  │   Pipeline   │───▶│   Pipeline   │───▶│    (Embeddings)      │  │
│  └──────────────┘    └──────────────┘    └──────────────────────┘  │
│         │                   │                       │               │
│    Documents           Chunks +                 Indexed             │
│                       Embeddings               Vectors              │
│                                                     │               │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐  │
│  │    Query     │    │  Retrieval   │    │   Context Assembly   │  │
│  │  Processing  │───▶│   Engine     │───▶│   + Generation       │  │
│  └──────────────┘    └──────────────┘    └──────────────────────┘  │
│         │                   │                       │               │
│    User Query          Top-K Chunks            LLM Response         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

## Document Ingestion Pipeline

### Document Processing Steps

```text
Raw Documents
      │
      ▼
┌─────────────┐
│   Extract   │ ← PDF, HTML, DOCX, Markdown
│   Content   │
└─────────────┘
      │
      ▼
┌─────────────┐
│   Clean &   │ ← Remove boilerplate, normalize
│  Normalize  │
└─────────────┘
      │
      ▼
┌─────────────┐
│   Chunk     │ ← Split into retrievable units
│  Documents  │
└─────────────┘
      │
      ▼
┌─────────────┐
│  Generate   │ ← Create vector representations
│ Embeddings  │
└─────────────┘
      │
      ▼
┌─────────────┐
│   Store     │ ← Persist vectors + metadata
│  in Index   │
└─────────────┘
```

## Chunking Strategies

### Strategy Comparison

| Strategy | Description | Best For | Chunk Size |
| -------- | ----------- | -------- | ---------- |
| **Fixed-size** | Split by token/character count | Simple documents | 256-512 tokens |
| **Sentence-based** | Split at sentence boundaries | Narrative text | Variable |
| **Paragraph-based** | Split at paragraph boundaries | Structured docs | Variable |
| **Semantic** | Split by topic/meaning | Long documents | Variable |
| **Recursive** | Hierarchical splitting | Mixed content | Configurable |
| **Document-specific** | Custom per doc type | Specialized (code, tables) | Variable |

### Chunking Decision Tree

```text
What type of content?
├── Code
│   └── AST-based or function-level chunking
├── Tables/Structured
│   └── Keep tables intact, chunk surrounding text
├── Long narrative
│   └── Semantic or recursive chunking
├── Short documents (<1 page)
│   └── Whole document as chunk
└── Mixed content
    └── Recursive with type-specific handlers
```

### Chunk Overlap

```text
Without Overlap:
[Chunk 1: "The quick brown"] [Chunk 2: "fox jumps over"]
                             ↑
               Information lost at boundary

With Overlap (20%):
[Chunk 1: "The quick brown fox"]
                    [Chunk 2: "brown fox jumps over"]
                         ↑
              Context preserved across boundaries
```

**Recommended overlap:** 10-20% of chunk size

### Chunk Size Trade-offs

```text
Smaller Chunks (128-256 tokens)        Larger Chunks (512-1024 tokens)
├── More precise retrieval             ├── More context per chunk
├── Less context per chunk             ├── May include irrelevant content
├── More chunks to search              ├── Fewer chunks to search
├── Better for factoid Q&A             ├── Better for summarization
└── Higher retrieval recall            └── Higher retrieval precision
```

## Embedding Models

### Model Comparison

| Model | Dimensions | Context | Strengths |
| ----- | ---------- | ------- | --------- |
| **OpenAI text-embedding-3-large** | 3072 | 8K | High quality, expensive |
| **OpenAI text-embedding-3-small** | 1536 | 8K | Good quality/cost ratio |
| **Cohere embed-v3** | 1024 | 512 | Multilingual, fast |
| **BGE-large** | 1024 | 512 | Open source, competitive |
| **E5-large-v2** | 1024 | 512 | Open source, instruction-tuned |
| **GTE-large** | 1024 | 512 | Alibaba, good for Chinese |
| **Sentence-BERT** | 768 | 512 | Classic, well-understood |

### Embedding Selection

```text
Need best quality, cost OK?
├── Yes → OpenAI text-embedding-3-large
└── No
    └── Need self-hosted/open source?
        ├── Yes → BGE-large or E5-large-v2
        └── No
            └── Need multilingual?
                ├── Yes → Cohere embed-v3
                └── No → OpenAI text-embedding-3-small
```

### Embedding Optimization

| Technique | Description | When to Use |
| --------- | ----------- | ----------- |
| **Matryoshka embeddings** | Truncatable to smaller dims | Memory-constrained |
| **Quantized embeddings** | INT8/binary embeddings | Large-scale search |
| **Instruction-tuned** | Prefix with task instruction | Specialized retrieval |
| **Fine-tuned embeddings** | Domain-specific training | Specialized domains |

## Retrieval Strategies

### Dense Retrieval (Semantic Search)

```text
Query: "How to deploy containers"
         │
         ▼
    ┌─────────┐
    │ Embed   │
    │ Query   │
    └─────────┘
         │
         ▼
    ┌─────────────────────────────────┐
    │ Vector Similarity Search        │
    │ (Cosine, Dot Product, L2)       │
    └─────────────────────────────────┘
         │
         ▼
    Top-K semantically similar chunks
```

### Sparse Retrieval (BM25/TF-IDF)

```text
Query: "Kubernetes pod deployment YAML"
         │
         ▼
    ┌─────────┐
    │Tokenize │
    │ + Score │
    └─────────┘
         │
         ▼
    ┌─────────────────────────────────┐
    │ BM25 Ranking                    │
    │ (Term frequency × IDF)          │
    └─────────────────────────────────┘
         │
         ▼
    Top-K lexically matching chunks
```

### Hybrid Search (Best of Both)

```text
Query ──┬──▶ Dense Search ──┬──▶ Fusion ──▶ Final Ranking
        │                   │      │
        └──▶ Sparse Search ─┘      │
                                   │
        Fusion Methods:            ▼
        • RRF (Reciprocal Rank Fusion)
        • Linear combination
        • Learned reranking
```

### Reciprocal Rank Fusion (RRF)

```text
RRF Score = Σ 1 / (k + rank_i)

Where:
- k = constant (typically 60)
- rank_i = rank in each retrieval result

Example:
Doc A: Dense rank=1, Sparse rank=5
RRF(A) = 1/(60+1) + 1/(60+5) = 0.0164 + 0.0154 = 0.0318

Doc B: Dense rank=3, Sparse rank=1
RRF(B) = 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323

Result: Doc B ranks higher (better combined relevance)
```

## Multi-Stage Retrieval

### Two-Stage Pipeline

```text
┌─────────────────────────────────────────────────────────┐
│ Stage 1: Recall (Fast, High Recall)                     │
│ • ANN search (HNSW, IVF)                                │
│ • Retrieve top-100 candidates                           │
│ • Latency: 10-50ms                                      │
└─────────────────────────────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────