chunking-strategies

What this skill does

# Chunking Strategies

**Purpose:** Provide production-ready document chunking implementations, benchmarking tools, and strategy selection guidance for RAG pipelines.

**Activation Triggers:**
- Implementing document chunking for RAG
- Optimizing chunk size and overlap
- Comparing different chunking strategies
- Benchmarking chunking performance
- Processing different document types (markdown, code, PDFs)
- Evaluating retrieval quality with different chunk strategies

**Key Resources:**
- `scripts/chunk-fixed-size.py` - Fixed-size chunking implementation
- `scripts/chunk-semantic.py` - Semantic chunking with paragraph preservation
- `scripts/chunk-recursive.py` - Recursive chunking for hierarchical documents
- `scripts/benchmark-chunking.py` - Benchmark and compare chunking strategies
- `templates/chunking-config.yaml` - Chunking configuration template
- `templates/custom-splitter.py` - Template for custom chunking logic
- `examples/chunk-markdown.py` - Markdown-specific chunking
- `examples/chunk-code.py` - Source code chunking
- `examples/chunk-pdf.py` - PDF document chunking

## Chunking Strategy Overview

### Strategy Selection Guide

**Fixed-Size Chunking:**
- Best for: Uniform documents, simple content, consistent structure
- Pros: Fast, predictable, simple implementation
- Cons: May split semantic units, no context awareness
- Use when: Speed matters more than semantic coherence

**Semantic Chunking:**
- Best for: Natural language documents, articles, books
- Pros: Preserves semantic boundaries, better context
- Cons: Slower, variable chunk sizes
- Use when: Content has clear paragraph/section structure

**Recursive Chunking:**
- Best for: Hierarchical documents, technical docs, code
- Pros: Preserves structure, handles nested content
- Cons: Most complex, requires structure detection
- Use when: Documents have clear hierarchical organization

**Sentence-Based Chunking:**
- Best for: Q&A pairs, chatbots, precise retrieval
- Pros: Natural boundaries, good for citations
- Cons: Small chunks may lack context
- Use when: Need precise attribution and citations

## Implementation Scripts

### 1. Fixed-Size Chunking

**Script:** `scripts/chunk-fixed-size.py`

**Usage:**
```bash
python scripts/chunk-fixed-size.py \
  --input document.txt \
  --chunk-size 1000 \
  --overlap 200 \
  --output chunks.json
```

**Parameters:**
- `chunk-size`: Number of characters per chunk (default: 1000)
- `overlap`: Character overlap between chunks (default: 200)
- `split-on`: Split on sentences, words, or characters (default: sentences)

**Best Practices:**
- Use 500-1000 character chunks for most RAG applications
- Set overlap to 10-20% of chunk size
- Split on sentences for better coherence

### 2. Semantic Chunking

**Script:** `scripts/chunk-semantic.py`

**Usage:**
```bash
python scripts/chunk-semantic.py \
  --input document.txt \
  --max-chunk-size 1500 \
  --output chunks.json
```

**How it works:**
1. Detects natural boundaries (paragraphs, headings, line breaks)
2. Groups content while respecting max chunk size
3. Preserves semantic units (paragraphs stay together)
4. Adds context headers for nested sections

**Best for:** Articles, blog posts, documentation, books

### 3. Recursive Chunking

**Script:** `scripts/chunk-recursive.py`

**Usage:**
```bash
python scripts/chunk-recursive.py \
  --input document.md \
  --chunk-size 1000 \
  --separators '["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]' \
  --output chunks.json
```

**How it works:**
1. Tries to split on first separator (e.g., ## headings)
2. If chunks still too large, recursively splits on next separator
3. Continues until all chunks are within size limit
4. Preserves hierarchical context

**Separator hierarchy examples:**
- **Markdown:** `["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]`
- **Python:** `["\\nclass ", "\\ndef ", "\\n\\n", "\\n", " "]`
- **General:** `["\\n\\n", "\\n", ". ", " "]`

**Best for:** Structured documents, source code, technical manuals

### 4. Benchmark Chunking Strategies

**Script:** `scripts/benchmark-chunking.py`

**Usage:**
```bash
python scripts/benchmark-chunking.py \
  --input document.txt \
  --strategies fixed,semantic,recursive \
  --chunk-sizes 500,1000,1500 \
  --output benchmark-results.json
```

**Metrics Evaluated:**
- **Processing time:** Speed of chunking
- **Chunk count:** Total chunks generated
- **Chunk size variance:** Consistency of chunk sizes
- **Context preservation:** Semantic unit integrity (scored)
- **Retrieval quality:** Simulated query performance

**Output:**
```json
{
  "fixed-1000": {
    "time_ms": 45,
    "chunk_count": 127,
    "avg_size": 982,
    "size_variance": 12.3,
    "context_score": 0.72
  },
  "semantic-1000": {
    "time_ms": 156,
    "chunk_count": 114,
    "avg_size": 1087,
    "size_variance": 234.5,
    "context_score": 0.91
  }
}
```

## Configuration Template

**Template:** `templates/chunking-config.yaml`

**Complete configuration:**
```yaml
chunking:
  # Global defaults
  default_strategy: semantic
  default_chunk_size: 1000
  default_overlap: 200

  # Strategy-specific configs
  strategies:
    fixed_size:
      chunk_size: 1000
      overlap: 200
      split_on: sentence  # sentence, word, character

    semantic:
      max_chunk_size: 1500
      min_chunk_size: 200
      preserve_paragraphs: true
      add_headers: true  # Include section headers

    recursive:
      chunk_size: 1000
      overlap: 100
      separators:
        markdown: ["\\n## ", "\\n### ", "\\n\\n", "\\n", " "]
        code: ["\\nclass ", "\\ndef ", "\\n\\n", "\\n", " "]
        text: ["\\n\\n", ". ", " "]

  # Document type mappings
  document_types:
    ".md": semantic
    ".py": recursive
    ".txt": fixed_size
    ".pdf": semantic
```

## Custom Splitter Template

**Template:** `templates/custom-splitter.py`

**Create your own chunking logic:**
```python
from typing import List, Dict
import re

class CustomChunker:
    def __init__(self, chunk_size: int = 1000, overlap: int = 200):
        self.chunk_size = chunk_size
        self.overlap = overlap

    def chunk(self, text: str, metadata: Dict = None) -> List[Dict]:
        """
        Implement custom chunking logic here.

        Returns:
            List of chunks with metadata:
            [
                {
                    "text": "chunk content",
                    "metadata": {
                        "chunk_id": 0,
                        "source": "document.txt",
                        "start_char": 0,
                        "end_char": 1000
                    }
                }
            ]
        """
        chunks = []

        # Your custom chunking logic here
        # Example: Split on custom pattern
        sections = self._split_sections(text)

        for i, section in enumerate(sections):
            chunks.append({
                "text": section,
                "metadata": {
                    "chunk_id": i,
                    "source": metadata.get("source", "unknown"),
                    "chunk_size": len(section)
                }
            })

        return chunks

    def _split_sections(self, text: str) -> List[str]:
        # Implement your splitting logic
        pass
```

## Document-Specific Examples

### Markdown Chunking

**Example:** `examples/chunk-markdown.py`

**Features:**
- Preserves heading hierarchy
- Keeps code blocks together
- Maintains list structure
- Adds parent section context to chunks

**Usage:**
```bash
python examples/chunk-markdown.py README.md --output readme-chunks.json
```

### Code Chunking

**Example:** `examples/chunk-code.py`

**Features:**
- Splits on class and function boundaries
- Preserves complete functions
- Includes docstrings with implementations
- Language-aware separator selection

**Supported languages:** Python, JavaScript, TypeScript, Java, Go

**Usage:**
```bash
python examples/chunk-code.py src/main.py --language python --output code-chunks.json
```

### PDF Chunking

**Example:** `examples/chunk-pdf.p