source-verify

What this skill does

# Source Verification

Verify that citations in a manuscript are real, accurate, and that quoted text actually appears in the source. Operates as a domain-specific audit-fix-loop: extract citations, run checks, score, fix, re-check. (See the `audit-fix-loop` skill for the canonical doctrine — auditor≠fixer, substrate-gate, anti-grind; this skill's substrate is citation/quote resolution.)

**Announce:** "Using source-verify to check citations against Paperpile and source documents."

## What This Skill Checks

```
┌──────────────────────────────────────────────────────────┐
│  CHECK 1: EXISTENCE (mechanical)                          │
│  Does this cited work exist in paperpile.bib?            │
│  → grep paperpile.bib for author + title + year          │
│  → If not found: flag as UNVERIFIED (may still exist)    │
└──────────────────────────────────────────────────────────┘
                        │
                  Exists in bib?
                        ▼
┌──────────────────────────────────────────────────────────┐
│  CHECK 2: ACCURACY (mechanical)                           │
│  Are the citation fields correct?                        │
│  → Compare volume, issue, pages, year against bibtex     │
│  → Flag any mismatches as FIELD_ERROR                    │
└──────────────────────────────────────────────────────────┘
                        │
                  Fields correct?
                        ▼
┌──────────────────────────────────────────────────────────┐
│  CHECK 3: QUOTE VERIFICATION (outsourced to RAG)          │
│  Does the quoted text appear in the source?              │
│  → readwise chat: "Verify this exact quote from [Author] │
│  → rga against downloaded PDF (fallback)                 │
│  → NLM generate-chat against notebook (fallback)         │
│  → Flag QUOTE_NOT_FOUND or QUOTE_MISMATCH               │
└──────────────────────────────────────────────────────────┘
                        │
                  Quotes verified?
                        ▼
┌──────────────────────────────────────────────────────────┐
│  CHECK 4: CLAIM GROUNDING (outsourced to RAG)             │
│  Does the cited source actually support the claim?       │
│  → readwise chat or NLM generate-chat                    │
│  → These systems answer ONLY from source text            │
│  → Flag UNSUPPORTED or CONTRADICTED                      │
└──────────────────────────────────────────────────────────┘
```

**Checks 1-2** run on every invocation (fast, mechanical). **Check 3** runs when the footnote contains a direct quote. **Check 4** runs only when the user explicitly requests claim grounding.

## Independence Architecture

The key insight: **verification must use external ground truth, never the agent's own memory.** Two systems provide this:

```
┌─────────────────────────────────────────────────────────┐
│  VERIFIER HIERARCHY — QUOTES (try in order)              │
│                                                          │
│  1. Readwise highlights — User highlighted the passage.  │
│     readwise-custom highlights --search "quote fragment"  │
│     If found: verified against actual source text.       │
│     Fastest — no download, no LLM.                       │
│                                                          │
│  2. rga (local)    — Download PDF from Drive, search.    │
│     Deterministic text search inside PDFs.               │
│     No pdftotext needed — rga extracts text internally.  │
│                                                          │
│  3. NLM chat       — Add paper to a verification         │
│     notebook, then query.                                │
│     "Find this passage in [source]: '…'"                │
│     Grounded in NLM's ingested sources.                  │
│     Best for: OCR issues, paraphrased quotes.            │
│                                                          │
│  NEVER: Agent's own memory/training data.                │
│  That is the hallucination source, not a verifier.       │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│  VERIFIER — CLAIM GROUNDING                              │
│                                                          │
│  NLM chat only — requires semantic understanding.        │
│  Readwise highlights are too fragmentary for claims.     │
│  rga is too literal for "does source support claim?"     │
└─────────────────────────────────────────────────────────┘
```

**Why Readwise first for quotes:** The user highlights important passages while reading. If a quote in the manuscript matches a Readwise highlight, it was captured directly from the source — strongest possible verification, zero cost.

**Why NLM for claims:** Claim grounding requires understanding what the source argues, not just matching strings. NLM chat answers from ingested full text with semantic comprehension.

## Scope

| Source Type | Checks Available | Ground Truth |
|-------------|-----------------|-------------|
| **Journal articles** | 1, 2, 3, 4 | `paperpile.bib` + Drive PDFs + Readwise highlights |
| **Books / book chapters** | 1, 2, 3, 4 | `paperpile.bib` + Drive PDFs |
| **Working papers** | 1, 2, 3, 4 | `paperpile.bib` + Drive PDFs |
| **SEC releases / regulations** | 1 (partial) | May be in bib as MISC entries |
| **Federal case citations** | 1 (existence only) | WRDS `fjc_litigation.civil` (13.5M cases) + `audit_corp_legal` |
| **Statutes** | NOT COVERED | No ground truth database wired up |

### Federal Case Verification via WRDS

The FJC Integrated Database (`fjc_litigation.civil`) contains all federal civil cases with plaintiff/defendant names, docket numbers, filing dates, districts, and nature-of-suit codes. Use it to verify that cited cases actually exist:

```bash
# Connect via SSH tunnel to WRDS
ssh wrds "echo \"SELECT plaintiff, defendant, docket, district, filedate
FROM fjc_litigation.civil
WHERE plaintiff ILIKE '%smith%' AND defendant ILIKE '%jones%'
AND filedate BETWEEN '2018-01-01' AND '2020-12-31'
LIMIT 10;\" | psql -h wrds-pgdata.wharton.upenn.edu -p 9737 -d wrds"
```

**Matching strategy for case cites:**
- Extract plaintiff and defendant names from the citation (e.g., "Smith v. Jones")
- Search `fjc_litigation.civil` with `ILIKE` on both fields
- Confirm year matches `filedate`
- For securities cases, filter `nos = 850` (nature of suit: securities/commodities)

**Audit Analytics** (`audit_corp_legal.f14_lit_legal_case`) provides corporate legal cases with settlement amounts, exposure dates, and docket numbers — useful for verifying securities litigation specifically.

**Limitations:**
- `casename` field is sparsely populated — match on `plaintiff`/`defendant` instead
- State court cases are NOT covered (federal only)
- Appellate case names may differ from trial court names
- Very recent cases may have a lag before appearing in the database

Statutes are not yet verifiable — no structured database wired up. Flag as `SKIPPED_NO_GROUND_TRUTH`.

<EXTREMELY-IMPORTANT>
## Iron Law: Mechanical Checks Before LLM Checks

**NEVER skip Checks 1-2 to jump straight to LLM-based verification.**

Mechanical checks against `paperpile.bib` are deterministic and free. They catch the most common hallucinations (invented papers, wrong volume/pages) without any LLM judgment. Running LLM checks without running mechanical checks first is wasting expensive calls on problems a grep would catch.

**Skipping the paperpile.bib check is NOT HELPFUL — the user publishes with unverified citations that may be hallucinated.**
</EXTREMELY-IMPORTANT>

## Step 0: Prerequisites

### Download paperpile.bib

The bibtex file lives on Google Drive and must be downloaded fresh each run:

```bash
# Download paperpile.bib from Drive
gws drive files get --account [email protected] \
  --params '{"fileId": "1yxibJLr1-kF_gcf3UA6QlU5ulXgR50kX", "alt": "media"}' \
  -o /tmp/paperpile.bib
```

**Always download fresh** — the user may have added new papers si