pdf-smart-extractor

What this skill does

# PDF Smart Extractor

**Extract and analyze large PDFs with minimal token usage**

---

## 🚨 AUTO-TRIGGER RULES (Read This First!)

**MANDATORY: You MUST use this skill when:**

1. **File Size Check:** User provides a `.pdf` file path
   - Run: `ls -lh <file_path>` or `stat -f%z <file_path>` (macOS) / `stat -c%s <file_path>` (Linux)
   - If size >1MB (1048576 bytes) → **AUTOMATICALLY suggest this plugin**
   - Do NOT attempt to read large PDFs directly

2. **Error Detection:** User encounters "PDF too large" or similar errors
   - **IMMEDIATELY** switch to this plugin
   - Explain: "This error occurs because the PDF exceeds limits. I'll use PDF Smart Extractor to solve this."

3. **Technical Documentation Keywords:** User mentions:
   - NIST, ISO, AWS, Azure, GCP, CIS, SANS documents
   - Compliance frameworks, security guides, technical manuals
   - **PROACTIVELY check file size and suggest this plugin**

4. **Token Efficiency:** User asks to analyze multi-page PDFs (>50 pages)
   - Suggest this plugin to save 90%+ tokens

**Example Auto-Trigger:**
```
User: "Can you analyze /docs/NIST.SP.800-161r1-upd1.pdf?"

You: [First, check file size]
$ ls -lh /docs/NIST.SP.800-161r1-upd1.pdf
-rw-r--r--  1 user  staff   3.3M Oct 20 10:30 /docs/NIST.SP.800-161r1-upd1.pdf

[Detect: 3.3MB > 1MB → Auto-trigger plugin]

"I notice this PDF is 3.3 MB. For efficient analysis, I'll use PDF Smart Extractor which will:
✓ Extract 100% of content locally (no token waste)
✓ Enable 12-100x faster querying
✓ Cache for instant future access

Starting extraction now..."
```

---

## When to Use This Skill

**IMPORTANT: This skill should be automatically suggested when:**
- User provides a PDF file path and the file is >1MB
- User mentions "PDF too large" or "can't read PDF"
- User tries to analyze technical documentation (NIST, ISO, AWS, Azure, GCP guides)
- User mentions token limits or context window issues with PDFs
- User wants to extract or analyze multi-page PDFs (>50 pages)

Use this skill when users need to:
- Extract content from large PDF documents (>1MB, >50 pages)
- Analyze PDFs that exceed LLM context windows
- Query specific sections of technical documents (NIST, ISO, AWS guides, etc.)
- Preserve 100% of PDF content while minimizing token consumption
- Build knowledge bases from PDF documentation
- Search PDFs for specific topics or keywords
- Overcome "PDF too large" errors

**Trigger phrases (explicit):**
- "extract this PDF"
- "analyze [PDF file]"
- "search [PDF] for [topic]"
- "what does [PDF] say about [topic]"
- "chunk this large PDF"
- "process NIST document"
- "read this PDF: /path/to/file.pdf"
- "can you analyze this technical document"

**Trigger phrases (implicit - auto-detect):**
- User provides path ending in `.pdf` and file size >1MB
- "PDF too large to read"
- "can't open this PDF"
- "this PDF won't load"
- "help me with this NIST/ISO/AWS/compliance document"
- "extract information from [large document]"
- "I have a big PDF file"

**Auto-detection logic:**
When user provides a file path:
1. Check if file extension is `.pdf`
2. Check file size using `ls -lh` or `stat`
3. If size >1MB, proactively suggest: "This PDF is X MB. I can use PDF Smart Extractor to process it efficiently with 100x less tokens. Would you like me to extract and chunk it?"

## Core Capabilities

### 1. Local PDF Extraction (Zero LLM Involvement)
- Extracts 100% of PDF content using PyMuPDF
- No LLM calls during extraction - fully local processing
- Preserves metadata, table of contents, and document structure
- Caches extracted content for instant reuse

### 2. Semantic Chunking
- Splits text at intelligent boundaries (chapters, sections, paragraphs)
- Preserves context and meaning across chunks
- Target chunk size: ~2000 tokens (configurable)
- 100% content preservation guaranteed

### 3. Efficient Querying
- Search chunks by keywords or topics
- Load only relevant chunks (12-25x token reduction)
- Ranked results by relevance
- Combine multiple chunks as needed

### 4. Persistent Caching
- One-time extraction per PDF
- Instant access to cached content
- File hash verification for integrity
- Automatic cache management

## Workflow

### Phase 1: Extract PDF (One-Time Setup)
```python
python scripts/extract_pdf.py /path/to/document.pdf
```

**What happens:**
- Reads entire PDF locally
- Extracts text, metadata, table of contents
- Saves to `~/.claude-pdf-cache/{cache_key}/`
- Returns cache key for future queries

**Output:**
- `full_text.txt` - Complete document text
- `pages.json` - Structured page data
- `metadata.json` - PDF metadata
- `toc.json` - Table of contents (if available)
- `manifest.json` - Extraction statistics

### Phase 2: Chunk Content (Semantic Organization)
```python
python scripts/semantic_chunker.py {cache_key}
```

**What happens:**
- Detects semantic boundaries (chapters, sections, paragraphs)
- Splits text at intelligent boundaries
- Creates ~2000 token chunks
- Preserves 100% of content

**Output:**
- `chunks.json` - Chunk index with metadata
- `chunks/chunk_0000.txt` - Individual chunk files
- Statistics: total chunks, token distribution, preservation rate

### Phase 3: Query Content (Efficient Retrieval)
```python
python scripts/query_pdf.py search {cache_key} "supply chain security"
```

**What happens:**
- Searches chunk index for relevant content
- Ranks results by relevance
- Returns only matching chunks
- Displays token counts for transparency

**Output:**
- List of matching chunks with previews
- Relevance scores
- Total tokens required (vs. full document)

## Usage Examples

### Example 1: Large NIST Document

**User Request:** "Extract and analyze NIST SP 800-161r1 for supply chain incident response procedures"

**Your Workflow:**

1. **Extract PDF (one-time):**
```bash
python scripts/extract_pdf.py /path/to/NIST.SP.800-161r1-upd1.pdf
```
Output: `Cache key: NIST.SP.800-161r1-upd1_a1b2c3d4e5f6`

2. **Chunk content:**
```bash
python scripts/semantic_chunker.py NIST.SP.800-161r1-upd1_a1b2c3d4e5f6
```
Output: Created 87 chunks, 98.7% content preservation

3. **Search for relevant sections:**
```bash
python scripts/query_pdf.py search NIST.SP.800-161r1-upd1_a1b2c3d4e5f6 "supply chain incident response"
```
Output:
- Chunk 23 - "Supply Chain Risk Management" (relevance: 87%, 1,850 tokens)
- Chunk 45 - "Incident Response in C-SCRM" (relevance: 72%, 2,010 tokens)
- Total: 3,860 tokens (vs. 48,000 for full document = 12.4x reduction)

4. **Retrieve specific chunks:**
```bash
python scripts/query_pdf.py get NIST.SP.800-161r1-upd1_a1b2c3d4e5f6 23
```
Output: Full content of chunk 23

5. **Provide context to user:**
"Based on NIST SP 800-161r1, supply chain incident response involves... [use chunk content]"

### Example 2: Multiple Related Queries

**User Request:** "I need to understand OT security incidents from NIST SP 800-82r3"

**Your Workflow:**

1. **Extract (one-time):**
```bash
python scripts/extract_pdf.py /path/to/NIST.SP.800-82r3.pdf
```

2. **Chunk:**
```bash
python scripts/semantic_chunker.py NIST.SP.800-82r3_x7y8z9
```

3. **First query - Overview:**
```bash
python scripts/query_pdf.py search NIST.SP.800-82r3_x7y8z9 "OT security overview"
```

4. **Second query - Incidents:**
```bash
python scripts/query_pdf.py search NIST.SP.800-82r3_x7y8z9 "incident response ICS"
```

5. **Third query - Specific threat:**
```bash
python scripts/query_pdf.py search NIST.SP.800-82r3_x7y8z9 "ransomware operational technology"
```

**Result:** Each query loads only relevant chunks (~2-4 chunks, ~5,000 tokens) instead of entire 8.2MB document (120,000+ tokens)

### Example 3: Table of Contents Navigation

**User Request:** "Show me the structure of this AWS security guide"

**Your Workflow:**

1. **Extract PDF:**
```bash
python scripts/extract_pdf.py aws-security-guide.pdf
```

2. **Get TOC:**
```bash
python scripts/query_pdf.py toc aws-security-guide_abc123
```

Output:
```
Chapter 1: Introduction (page 1)
  1.1 Security Fundamentals (page 3)
  1.2 Shared R