pdf-form-filler

What this skill does

# PDF Form Filler Skill

You are an expert PDF form filling system. You fill any PDF with perfect field accuracy using two distinct modes:

1. **AcroForm mode** — PDFs with fillable fields: radial spatial identity, scanline context, Gemini vision, server-side validation
2. **Flat PDF mode** — PDFs with NO fillable fields: content stream geometry scanning, floor/ceiling detection, FreeText annotations with appearance streams

## Quick Reference

**CRITICAL FILES** (read these first when debugging):
- `src/hooks/usePdfFormFields.js` — Field extraction + spatial labels
- `src/utils/scanlineAccumulator.js` — Radial identity + bbox scoring
- `src/utils/formFieldsCsv.js` — CSV generation (13 columns)
- `server/src/routes/ai.ts` — Server: analyze-form, fill validation, prompt builder
- `src/validator.ts` — Server-side validation engine (5 rules)
- `src/form-state.ts` — Form state simulator (buildFormState, applyFields)
- `src/eval-runner.ts` — Eval assertion runner + intent parser

**SCRIPTS** (run from `PDF-Filler-Skill/scripts/`):
- `verify-field-map.ts` — Validate field mapping against ground truth
- `diagnose-field.ts` — Debug why a specific field maps wrong
- `dump-field-context.ts` — Dump all spatial context for every field
- `eval-fill-accuracy.ts` — Run automated fill accuracy eval
- `live-fill.ts` — End-to-end pipeline: prompt → intent → validate → fill JSON
- `flat-pdf-fill.py` — Flat PDF scanner + annotation filler (no AcroForm required)

**TESTS** (115 tests, 686 assertions):
- `__tests__/ground-truth.test.ts` — 27 tests: field map integrity
- `__tests__/validator.test.ts` — 25 tests: all 5 validation rules
- `__tests__/form-state.test.ts` — 17 tests: state management
- `__tests__/e2e-scenarios.test.ts` — 46 tests: all 6 eval scenarios + filing status matrix

For detailed architecture, see `architecture.md`.
For field extraction internals, see `field-extraction-guide.md`.
For validation rules, see `validation-rules.md`.
For system prompt construction, see `system-prompt-reference.md`.
For flat PDF filling technique, see `docs/flat-pdf-filling-guide.md`.

## How Form Filling Works

### Pipeline Overview

```
PDF opened in StickyApp
  |
  v
[1. EXTRACT] pdfjs-dist: getFieldObjects() + getTextContent()
  |           -> AcroForm fields (name, type, rect, options, maxLen, tooltip)
  |           -> Text layout (every character with x,y coordinates)
  |
  v
[2. ENRICH]  Three-layer spatial label matching:
  |           Layer 1: Heuristic directional (findLabelLeft/Above/Below/Right)
  |           Layer 2: Radial identity (ray cast + fragment clustering + line joining)
  |           Layer 3: Scanline accumulator (reading-order context + section detection)
  |           Cross-validation: bboxAdjacencyScore() confirms or overrides matches
  |
  v
[3. ANALYZE] Gemini vision (optional, auto-triggers if no cached template):
  |           -> Inject FIELD_IDs into PDF via pdf-lib
  |           -> Send modified PDF to Gemini page-by-page
  |           -> Parse: description, section, lineNumber, semanticType, format,
  |              calculation, dependencies, confidence
  |           -> Multi-pass: follow-up for missed fields
  |           -> Cache results as template in IndexedDB
  |
  v
[4. PROMPT]  Build AI system prompt:
  |           IF vision template: 13-column CSV (compact, semantic)
  |           ELSE: XML + Field Lookup Index + Text Search Index
  |
  v
[5. FILL]    AI calls fill_form_fields({ fields: {name: value}, reason })
  |           -> Server validates: unknown names, read-only, checkbox bool,
  |              dropdown options
  |           -> Valid fields emitted via SSE to client
  |           -> Validation errors returned to AI for retry
  |
  v
[6. EXPORT]  pdf-lib: write values -> flatten form -> overlay annotations -> download
```

### Mode 2: Flat PDF Pipeline (No AcroForm Fields)

For PDFs like the IRS W-9 that have NO fillable fields — just static drawings:

```
PDF with no AcroForm fields
  |
  v
[1. SCAN]    Parse content stream for geometric primitives:
  |           -> `re` operator: rectangles (checkboxes, digit boxes, borders)
  |           -> `q 1 0 0 1 TX TY cm ... S Q`: line segments with transforms
  |
  v
[2. CLASSIFY] Rectangle size → element type:
  |           -> 8×8 pt = checkbox
  |           -> 14.4×24 pt = digit input box
  |           -> Gaps between digit boxes = dash separators
  |           -> Pattern 3-2-4 = SSN (XXX-XX-XXXX)
  |           -> Pattern 2-7 = EIN (XX-XXXXXXX)
  |
  v
[3. DETECT]  Floor/ceiling detection:
  |           -> Extract text label positions via visitor pattern
  |           -> For each label, scan DOWN through h_lines to find floor (underline)
  |           -> For each label, scan UP to find ceiling (line above)
  |           -> Text sits ON the floor line, not below it
  |
  v
[4. PLACE]   FreeText annotations with appearance streams:
  |           -> Build /AP dictionary with PDF content stream (BT/Tf/Td/Tj/ET)
  |           -> Register /Helv font in /Resources
  |           -> Set /F=4 (print flag), /BS /W=0 (no border)
  |           -> Without /AP, annotations are INVISIBLE in most viewers
  |
  v
[5. WRITE]   pypdf writes annotations to output PDF
```

#### Content Stream Scanning

PDF pages are sequences of drawing commands. The `re` operator draws rectangles:
```
417.6 396.0 14.4 24.0 re    → digit box at (417.6, 396.0), 14.4w × 24h
73.0 603.7 8.0 8.0 re       → checkbox at (73.0, 603.7), 8w × 8h
```

Line segments use transformation matrices inside save/restore pairs:
```
q 1 0 0 1 543.35 587.97 cm   → translate to (543.35, 587.97)
0 0 m                         → moveto (0,0) in local coords
32.9 0 l                      → lineto (32.9, 0) — horizontal line
S                             → stroke
Q                             → restore state
```
Real position: horizontal line at y=587.97 from x=543.35 to x=576.25.

#### Floor Detection Algorithm

From a label position, scan outward then down to find the underline:

1. **Start** at label (x, y) — e.g., "Exempt payee code (if any)" at (457.6, 592.5)
2. **Scan down** through all horizontal lines where:
   - `line.y < label.y` (below the label)
   - `line.y > label.y - 40` (within 40pt)
   - Line overlaps horizontally with the label's x ± tolerance
3. **Pick nearest** (highest y = closest floor below)
4. **Place text** with annotation rect bottom AT the floor y-coordinate

Result: text sits precisely on the underline, exactly where a human would write.

#### Appearance Streams (Critical)

pypdf's `FreeText()` helper creates annotations WITHOUT `/AP` (appearance dictionary). Most PDF viewers ignore annotations that lack `/AP`. You MUST build it manually:

```python
# The appearance stream is a mini PDF content stream
ap_content = "BT\n/Helv 10 Tf\n0 g\n2 1.0 Td\n(5) Tj\nET"

ap_stream["/Type"] = "/XObject"
ap_stream["/Subtype"] = "/Form"
ap_stream["/BBox"] = [0, 0, width, height]
ap_stream["/Resources"] = {"/Font": {"/Helv": <Helvetica font dict>}}

annot["/AP"] = {"/N": ap_stream}   # /N = Normal appearance
annot["/DA"] = "/Helv 10 Tf 0 g"  # Default appearance fallback
annot["/F"] = 4                     # Print flag
annot["/BS"] = {"/W": 0}           # No border
```

### The Three Enrichment Layers

**Layer 1 — Heuristic Directional** (`usePdfFormFields.js`):
- For each field, search LEFT/ABOVE/BELOW/RIGHT for nearest text
- Scoring: `distance * bandOverlapBonus` (bbox Y-band overlap breaks ties)
- Fast but fails on dense forms where widget Y-offset (0.008) exceeds half the row spacing

**Layer 2 — Radial Identity** (`scanlineAccumulator.js`):
- Each field independently casts rays in 4 cardinal directions
- Collects ALL text fragments within ray band, clusters by Y (tolerance: 0.004)
- Joins fragments into full lines: "Your " + "first " + "name" -> "Your first name"
- LEFT ray: Y band = 0.025, picks line with Y >= field center (accounts for widget offset)
- Cross-validated with `bboxAdjacencyScore(field, text, direction)`:
  - Score