document-parsers

What this skill does

# Document Parsers

**Purpose:** Autonomously parse and extract content from multiple document formats (PDF, DOCX, HTML, Markdown) using industry-standard libraries and AI-powered parsing tools.

**Activation Triggers:**
- Building RAG (Retrieval-Augmented Generation) pipelines
- Extracting text, tables, or metadata from documents
- Processing large document collections
- Converting documents to structured formats
- Handling complex PDFs with tables and layouts
- OCR for scanned documents
- Chunking documents for vector embeddings
- Building document search systems

**Key Resources:**
- `scripts/setup-llamaparse.sh` - Install and configure LlamaParse (AI-powered parsing)
- `scripts/setup-unstructured.sh` - Install Unstructured.io library
- `scripts/parse-pdf.py` - Functional PDF parser with multiple backend options
- `scripts/parse-docx.py` - DOCX document parser
- `scripts/parse-html.py` - HTML to structured text parser
- `templates/multi-format-parser.py` - Universal document parser template
- `templates/table-extraction.py` - Specialized table extraction template
- `examples/parse-research-paper.py` - Research paper parsing with citations
- `examples/parse-legal-document.py` - Legal document parsing with sections

## Parser Comparison & Selection Guide

### 1. LlamaParse (AI-Powered Premium)

**Best For:**
- Complex PDFs with tables, charts, and mixed layouts
- Scanned documents requiring OCR
- Documents where accuracy is critical
- Multi-column layouts and scientific papers
- Financial reports and invoices

**Pros:**
- AI-powered layout understanding
- Excellent table extraction accuracy
- Built-in OCR support
- Handles complex formatting
- Structured output (Markdown/JSON)

**Cons:**
- Requires API key (paid service)
- API rate limits
- Network dependency
- Slower than local parsers

**Documentation:** https://docs.cloud.llamaindex.ai/llamaparse

**Setup:**
```bash
./scripts/setup-llamaparse.sh
```

**Usage Pattern:**
```python
from llama_parse import LlamaParse

parser = LlamaParse(
    api_key="llx-...",
    result_type="markdown",  # or "text"
    language="en",
    verbose=True
)

documents = parser.load_data("document.pdf")
for doc in documents:
    print(doc.text)
```

### 2. Unstructured.io (Local Processing)

**Best For:**
- Batch processing many documents
- Multiple format support (PDF, DOCX, HTML, PPTX, Images)
- Local processing without API dependencies
- Structured element extraction
- Production RAG pipelines

**Pros:**
- Open-source and free
- Multi-format support
- Runs locally (no API keys)
- Good table detection
- Element-based chunking

**Cons:**
- Requires system dependencies (poppler, tesseract)
- Complex installation
- Less accurate than LlamaParse for complex layouts

**Documentation:** https://unstructured-io.github.io/unstructured/

**Setup:**
```bash
./scripts/setup-unstructured.sh
```

**Usage Pattern:**
```python
from unstructured.partition.auto import partition

elements = partition("document.pdf")
for element in elements:
    print(f"{element.category}: {element.text}")
```

### 3. PyPDF2 (Simple PDF Text Extraction)

**Best For:**
- Simple text-based PDFs
- Quick prototyping
- Metadata extraction
- PDF manipulation (merge, split)

**Pros:**
- Pure Python (no dependencies)
- Fast and lightweight
- Good for simple PDFs
- Active maintenance

**Cons:**
- Poor table extraction
- Struggles with complex layouts
- No OCR support
- Limited formatting preservation

**Documentation:** https://github.com/py-pdf/pypdf2

**Setup:**
```bash
pip install pypdf2
```

**Usage Pattern:**
```python
from PyPDF2 import PdfReader

reader = PdfReader("document.pdf")
for page in reader.pages:
    print(page.extract_text())
```

### 4. PDFPlumber (Advanced PDF Analysis)

**Best For:**
- Table extraction from PDFs
- PDF with tabular data
- Financial statements and reports
- Coordinate-based extraction

**Pros:**
- Excellent table extraction
- Visual debugging tools
- Coordinate-level control
- Metadata and layout info

**Cons:**
- Slower than PyPDF2
- Requires pdfminer.six dependency
- No OCR support

**Documentation:** https://github.com/jsvine/pdfplumber

**Setup:**
```bash
pip install pdfplumber
```

**Usage Pattern:**
```python
import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for page in pdf.pages:
        tables = page.extract_tables()
        text = page.extract_text()
```

### 5. python-docx (Word Documents)

**Best For:**
- Microsoft Word (.docx) documents
- Extracting paragraphs, tables, headers
- Document metadata
- Template-based document generation

**Pros:**
- Native DOCX support
- Preserves structure (paragraphs, tables, sections)
- Access to styles and formatting
- Can also write/modify DOCX

**Cons:**
- Only works with .docx (not .doc)
- Limited image extraction

**Documentation:** https://github.com/python-openxml/python-docx

**Setup:**
```bash
pip install python-docx
```

**Usage Pattern:**
```python
from docx import Document

doc = Document("document.docx")
for para in doc.paragraphs:
    print(para.text)
for table in doc.tables:
    for row in table.rows:
        print([cell.text for cell in row.cells])
```

## Decision Matrix

| Use Case | Recommended Parser | Alternative |
|----------|-------------------|-------------|
| Simple PDF text extraction | PyPDF2 | Unstructured |
| Complex PDFs with tables | LlamaParse | PDFPlumber |
| Scanned documents (OCR) | LlamaParse | Unstructured + Tesseract |
| Word documents (.docx) | python-docx | Unstructured |
| HTML to text | parse-html.py | Unstructured |
| Multi-format batch processing | Unstructured | Multi-format-parser |
| Table extraction | PDFPlumber | LlamaParse |
| Research papers | LlamaParse | Unstructured |
| Legal documents | LlamaParse | PDFPlumber |
| Production RAG pipeline | Unstructured | LlamaParse |

## Functional Scripts

### 1. Parse PDF (`scripts/parse-pdf.py`)

Command-line PDF parser supporting multiple backends:

```bash
# Using PyPDF2 (default)
python scripts/parse-pdf.py document.pdf

# Using PDFPlumber (better for tables)
python scripts/parse-pdf.py document.pdf --backend pdfplumber

# Using LlamaParse (AI-powered)
python scripts/parse-pdf.py document.pdf --backend llamaparse --api-key llx-...

# Output to file
python scripts/parse-pdf.py document.pdf --output output.txt

# Extract tables as JSON
python scripts/parse-pdf.py document.pdf --backend pdfplumber --tables-only --output tables.json
```

**Features:**
- Multiple backend support (PyPDF2, PDFPlumber, LlamaParse)
- Table extraction
- Metadata extraction
- Page range selection
- JSON/Text output formats

### 2. Parse DOCX (`scripts/parse-docx.py`)

Word document parser with structure preservation:

```bash
# Basic extraction
python scripts/parse-docx.py document.docx

# Extract with structure
python scripts/parse-docx.py document.docx --preserve-structure

# Extract tables only
python scripts/parse-docx.py document.docx --tables-only

# Output as JSON
python scripts/parse-docx.py document.docx --output output.json --format json
```

**Features:**
- Paragraph extraction with styles
- Table extraction
- Header/footer extraction
- Metadata (author, created date, etc.)
- Structured JSON output

### 3. Parse HTML (`scripts/parse-html.py`)

HTML to clean text converter:

```bash
# Basic HTML parsing
python scripts/parse-html.py document.html

# From URL
python scripts/parse-html.py https://example.com/article

# Preserve links
python scripts/parse-html.py document.html --preserve-links

# Extract specific selector
python scripts/parse-html.py document.html --selector "article.content"
```

**Features:**
- Clean text extraction (removes scripts, styles)
- Link preservation
- CSS selector support
- URL fetching
- Markdown output option

## Templates

### Multi-Format Parser (`templates/multi-format-parser.py`)

Universal parser handling multiple formats with automatic format detection:

```python
from multi_format_parser import MultiFormatParser

parser = MultiFormatParser(
    llamap