legal-red-team

What this skill does

> **⚠️ EXPERIMENTAL** — This skill is provided for educational and informational purposes only. It does NOT constitute legal advice. All responsibility for usage rests with the user. Consult qualified legal professionals before acting on any output.

# Legal Red Team

Production-ready adversarial verification framework for AI-generated legal content. Covers factual accuracy, citation validation, arithmetic checking, speculation detection, and distribution readiness scoring.

---

## Table of Contents

- [Verification Categories](#verification-categories)
- [Tools](#tools)
- [Six-Step Methodology](#six-step-methodology)
- [Severity Taxonomy](#severity-taxonomy)
- [Quality Score](#quality-score)
- [Known Hallucination Patterns](#known-hallucination-patterns)
- [Reference Guides](#reference-guides)
- [Workflows](#workflows)
- [Troubleshooting](#troubleshooting)
- [Success Criteria](#success-criteria)
- [Scope and Limitations](#scope-and-limitations)
- [Anti-Patterns](#anti-patterns)
- [Tool Reference](#tool-reference)

---

## Verification Categories

Every AI-generated legal document must be checked across 6 categories.

| # | Category | What to Check | Red Flags |
|---|----------|--------------|-----------|
| 1 | **Factual Accuracy** | Dates, references, numbers, entity names, timelines | Wrong effective dates, confused entity names, incorrect amounts |
| 2 | **Legal Authority Citations** | Primary/secondary sources, format, hierarchy, currency | Non-existent articles, wrong section numbers, outdated citations |
| 3 | **Arithmetic Validation** | Timelines, percentages, financial calculations, deadlines | Date math errors, percentage miscalculations, compounding mistakes |
| 4 | **Source Verification** | Verifiable claims, official sources, cross-referencing | Unverifiable assertions stated as fact, single-source claims |
| 5 | **Speculation Detection** | Opinion vs fact, uncertainty language, predictive claims | Predictions stated as certainty, guidance treated as binding law |
| 6 | **Disclaimer Adequacy** | Legal advice disclaimers, jurisdiction, date, professional consultation | Missing disclaimers, overly broad claims, no jurisdiction limits |

---

## Tools

### Legal Fact Checker

Scans legal text for verifiable claims and flags potential hallucination patterns.

```bash
# Check a legal document
python scripts/legal_fact_checker.py --input document.txt

# Check with JSON output
python scripts/legal_fact_checker.py --input memo.txt --json

# Check inline text
python scripts/legal_fact_checker.py --text "Under GDPR Article 83(5), fines can reach EUR 20 million..."

# Save verification report
python scripts/legal_fact_checker.py --input document.txt --output report.json
```

### Legal Quality Scorer

Scores legal document quality across all 6 verification categories.

```bash
# Score a document
python scripts/legal_quality_scorer.py --input document.txt

# Score with JSON output
python scripts/legal_quality_scorer.py --input document.txt --json

# Score with detailed breakdown
python scripts/legal_quality_scorer.py --input document.txt --verbose

# Save quality assessment
python scripts/legal_quality_scorer.py --input document.txt --output assessment.json
```

---

## Six-Step Methodology

### Step 1: Initial Review

Read the entire document with an adversarial mindset. For each claim, ask:

- Is this verifiable?
- Does this sound too specific to be generated without a source?
- Does this sound too confident for an uncertain area?

Mark every factual assertion, citation, date, number, and predictive statement.

### Step 2: Source Verification (ALWAYS Web Search)

For every verifiable claim, attempt to verify against official sources.

| Source Type | Verification Method | Examples |
|-------------|-------------------|---------|
| EU legislation | EUR-Lex official database | eur-lex.europa.eu |
| US federal law | congress.gov, govinfo.gov | Official code and statutes |
| US regulations | eCFR, Federal Register | ecfr.gov |
| UK legislation | legislation.gov.uk | Official statute database |
| Court decisions | Court databases, Westlaw, LexisNexis | Official reporters |
| Agency guidance | Agency official website | Direct download from .gov/.europa.eu |
| International treaties | UN Treaty Collection | treaties.un.org |

**Rule:** If a claim cannot be verified from an official source, flag it. Do not assume accuracy.

### Step 3: Arithmetic Verification

Check every calculation, date computation, and numerical claim.

| Check Type | Method |
|-----------|--------|
| Timeline calculations | Count days/months/years between stated dates |
| Percentage calculations | Recalculate from base figures |
| Financial computations | Verify arithmetic and compounding |
| Deadline calculations | Confirm against statutory text |
| Penalty ranges | Cross-check against statute |

### Step 4: Citation Validation

For every legal citation, verify:

| Element | Check |
|---------|-------|
| Source exists | Does the cited statute/article/section actually exist? |
| Content matches | Does the cited provision say what the document claims? |
| Citation format | Is the citation in correct format for the jurisdiction? |
| Currency | Is this the current, in-force version? |
| Hierarchy correct | Is the source characterized at the right authority level? |

### Step 5: Speculation Identification

Distinguish fact from opinion, certainty from prediction.

| Language Pattern | Classification | Action |
|-----------------|---------------|--------|
| "The law requires..." | Factual claim | Verify against statutory text |
| "Courts will likely..." | Speculation | Flag; add uncertainty qualifier |
| "It is recommended..." | Guidance | Verify source; clarify if binding |
| "Best practice suggests..." | Opinion | Label as opinion; cite source |
| "This means that..." | Interpretation | Flag if stated as fact without authority |
| "Companies must..." | Obligation claim | Verify statutory basis |

### Step 6: Disclaimer Review

Every AI-generated legal document must include:

| Required Element | Description |
|-----------------|-------------|
| Not legal advice | Clear statement that content is informational only |
| Jurisdiction limitations | Which jurisdictions are and are not covered |
| Date of preparation | When the content was prepared (law changes) |
| Professional consultation | Recommendation to consult qualified legal counsel |
| AI-generated disclosure | Statement that content was generated or assisted by AI |
| Accuracy limitations | Acknowledgment that verification is recommended |

---

## Severity Taxonomy

| Severity | Definition | Examples | Action |
|----------|-----------|---------|--------|
| **CRITICAL** | Factually wrong in a way that could cause legal harm | Wrong article number creating false obligation, incorrect penalty amount, non-existent legal requirement | Must fix before any distribution |
| **HIGH** | Materially misleading or unverifiable | Guidance stated as binding law, unverifiable timeline, confident but unsourced claim | Must fix or add prominent caveat |
| **MODERATE** | Imprecise or potentially confusing | Ambiguous language, minor date discrepancy, incomplete citation | Should fix; acceptable with caveat |
| **LOW** | Style or formatting issue | Citation format inconsistency, missing cross-reference, minor redundancy | Fix if time permits |

---

## Quality Score

| Score | Rating | Distribution Status | Criteria |
|-------|--------|-------------------|----------|
| **5/5** | Distribution Ready | Safe to distribute | Zero CRITICAL/HIGH issues; all citations verified; disclaimers complete |
| **4/5** | Minor Revisions | Safe after small fixes | Zero CRITICAL; 1-2 HIGH issues with clear fixes; most citations verified |
| **3/5** | Moderate Revisions | Needs work before distribution | Zero CRITICAL; 3+ HIGH issues; some unverified citations |
| **2/5** | Major Revisions | Not safe to distribute | 1+ CRITICAL issues; multiple HIGH issues; significant