paper-relevance-filter

What this skill does

# paper-relevance-filter

Decide whether a fetched paper belongs in this week's digest. Output is a per-paper decision (KEEP / DROP / REVIEW), a 0-100 score, and a one-line rationale that the user can audit.

## Workflow

```
- [ ] Step 1: Load relevance criteria + the watchlist + last-4-weeks kept-paper IDs
- [ ] Step 2: Score each paper on three axes (match, criteria, novelty)
- [ ] Step 3: Combine to a 0-100 score; map to KEEP / REVIEW / DROP via thresholds
- [ ] Step 4: Apply tie-breakers (cap output at requested max kept count)
- [ ] Step 5: Return decisions + a calibration summary
```

**Step 1 — Inputs**

The caller hands the skill:
- `papers`: list of normalized paper records (output of `fetch-preprint-recent` or `fetch-pubmed-recent`)
- `watchlist`: list of keywords/phrases (with optional weights — default weight 1.0)
- `criteria`: text from `relevance-criteria.md` describing what fits and what doesn't
- `prior_ids`: set of `id` values that appeared as KEEP in any of the last 4 digests (used for novelty)
- `max_kept`: target ceiling, e.g. 25 (the digest will not exceed this)

**Step 2 — Three-axis scoring**

For each paper, compute three sub-scores in [0, 1]:

**Axis 1 — Match strength (0-1)**
- 0.0 if no watchlist keyword appears in title or abstract
- 0.5 if a keyword appears once in the abstract only
- 0.7 if a keyword appears in the abstract more than once OR in title once
- 1.0 if a keyword appears in the title AND abstract, or multiple distinct keywords match

If keywords carry weights (some matter more than others), use the max weight among matched keywords as a multiplier capped at 1.0.

**Axis 2 — Criteria fit (0-1)**
This is the qualitative axis. Read the abstract against the relevance-criteria document. The criteria typically state:
- In-scope concepts (the field/method the user actually wants)
- Out-of-scope concepts (look-alikes the keyword filter might let through)
- Required minimums (e.g., "must report empirical results", "must be primary research, not commentary")

Score:
- 1.0 — clearly in-scope, meets minimums, no look-alike traps triggered
- 0.7 — in-scope but borderline (review article on the topic; short report; preprint with no methods detail in abstract)
- 0.4 — partially in-scope (touches the topic but the paper's main subject is elsewhere)
- 0.0 — out-of-scope or trips an explicit exclusion (e.g., "exclude pure-theory papers" and the abstract is pure theory)

If the criteria document is silent on a paper's territory, default to 0.7 and flag for REVIEW.

**Axis 3 — Novelty (0-1)**
- 1.0 if `id` is not in `prior_ids` and the title doesn't fuzzy-match any prior title
- 0.5 if a prior preprint version exists in `prior_ids` (e.g., same DOI prefix `10.1101/...` matched, this is a journal version) — KEEP-worthy but tag as "journal version of preprint covered YYYY-WW"
- 0.0 if exact `id` match in `prior_ids` (already covered)

Use normalized title (lowercase, strip punctuation, collapse whitespace) for fuzzy matching. A Levenshtein ratio > 0.9 against any prior title counts as a match.

**Step 3 — Combine and threshold**

```
score = 100 * (0.45 * match + 0.45 * criteria + 0.10 * novelty)
```

Match and criteria carry equal weight (a paper that mentions your keywords once but is wildly out-of-topic should not score higher than one that's deeply on-topic with a single mention). Novelty is a small finger on the scale — enough to demote already-covered work but not enough to drop a genuinely important journal-version-of-preprint update.

Decision thresholds (default; the calling agent may override):

| Score    | Decision | Notes                                                          |
| -------- | -------- | -------------------------------------------------------------- |
| 70-100   | KEEP     | Goes into the digest                                           |
| 50-69    | REVIEW   | Boundary cases — caller decides whether to escalate to user    |
| 0-49     | DROP     | Filtered out, reason logged in the dropped-papers section      |

Special-case override: if `novelty == 0.0` (already in a prior digest), force DROP regardless of score. The papers section may still list it as "already covered" for traceability.

**Step 4 — Tie-breakers when KEEP > max_kept**

When more papers score ≥ 70 than `max_kept`:

1. Re-score with stricter axis-2 thresholds (review articles drop from 0.7 to 0.4; partial-fit drops from 0.4 to 0.2). This is the cleanest tightening.
2. If still over, sort KEEPs by score descending, take the top `max_kept`, and demote the rest to REVIEW (not DROP — they were good enough; just couldn't fit). Surface this in the calibration summary.

Never demote to DROP what scored ≥ 70 unless explicitly forced.

**Step 5 — Return**

```json
{
  "decisions": [
    {
      "id": "10.1101/2026.05.07.123456",
      "decision": "KEEP",
      "score": 84,
      "axes": {"match": 0.9, "criteria": 1.0, "novelty": 1.0},
      "rationale": "Title + abstract hit 'protein language model' twice; in-scope (primary methods paper, empirical); novel.",
      "tags": []
    },
    {
      "id": "PMID:39000000",
      "decision": "KEEP",
      "score": 72,
      "axes": {"match": 1.0, "criteria": 0.7, "novelty": 0.5},
      "rationale": "Strong keyword match; review article (criteria penalty); journal version of preprint covered 2026-15.",
      "tags": ["journal-version-of:2026-15"]
    },
    {
      "id": "PMID:39111111",
      "decision": "DROP",
      "score": 31,
      "axes": {"match": 0.5, "criteria": 0.0, "novelty": 1.0},
      "rationale": "'protein language model' appears once in abstract but the paper is a clinical trial enrollment report — out of scope.",
      "tags": ["look-alike-trap"]
    }
  ],
  "calibration": {
    "kept": 17,
    "review": 3,
    "dropped": 84,
    "force_dropped_already_covered": 2,
    "demoted_for_cap": 0,
    "stricter_pass_applied": false
  }
}
```

## Common Patterns

**Pattern A — Strict weekly digest**: defaults above. Tight thresholds; max_kept=25.

**Pattern B — Catch-up over multiple weeks**: run per-week with the same prior_ids growing each iteration. Don't pool all 3 weeks of papers and filter once — you'll lose the historical-context signal.

**Pattern C — Topic deep-dive (user wants more, not less)**: relax max_kept to a high number (e.g. 100), keep thresholds, return the full ranked list. Only do this on explicit user request.

**Pattern D — Sanity-check the watchlist itself**: run with `prior_ids = []` and look at calibration.dropped. If the same theme keeps getting dropped for criteria reasons, the watchlist may be drifting away from intent.

## Guardrails

1. **Never decide based on title alone if the abstract is available.** Titles overstate; abstracts qualify.
2. **Never KEEP a paper whose abstract isn't returned.** Tag it REVIEW so the user can decide; an empty abstract is a signal something went wrong upstream, not a green light.
3. **Always explain the DROP.** A one-line rationale per dropped paper is non-negotiable — without it the user cannot audit the filter or notice systematic blind spots.
4. **Don't conflate "out of scope" with "low quality".** This skill judges fit, not quality. A high-quality paper outside scope is a DROP; a low-quality paper inside scope is a KEEP with a rationale flag.
5. **Don't auto-update the watchlist.** If you notice the criteria are pulling in a class of papers the watchlist doesn't anticipate, surface it in the calibration summary; the user owns the watchlist edit.
6. **Don't dedupe inside this skill.** The caller is responsible for cross-source dedupe (a bioRxiv preprint and its PubMed publication are *the same paper* and must be merged before this skill sees them, otherwise novelty scoring breaks).
7. **Don't use unbounded LLM judgment for axis 2.** Keep the criteria-fit decision anchored to the explicit `relevance-criteria.md` text. If something's not in the criteria, the answer is REVIEW with rationale "criteria silent" — not "I think it