result-to-claim

What this skill does

# Result-to-Claim Gate

> 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is
> verdict-bearing — it judges whether results support a claim. Re-running that
> verdict on a wall-clock timer adds no new signal (the verdict changes only
> when the *results* change, not when the clock ticks). What you actually want
> to schedule is the *external wait that precedes it* — experiments done → then
> run this gate **once**. See
> [`shared-references/external-cadence.md`](../shared-references/external-cadence.md).

Experiments produce numbers; this gate decides what those numbers *mean*. Collect results from available sources, get a Codex judgment, then auto-route based on the verdict.

## Context: $ARGUMENTS

## When to Use

- After a set of experiments completes (main results, not just sanity checks)
- Before committing to claims in a paper or review response
- When results are ambiguous and you need an objective second opinion

## Workflow

### Step 1: Collect Results

Gather experiment data from whatever sources are available in the project:

1. **W&B** (preferred): `wandb.Api().run("<entity>/<project>/<run_id>").history()` — metrics, training curves, comparisons
2. **EXPERIMENT_LOG.md**: full results table with baselines and verdicts
3. **EXPERIMENT_TRACKER.md**: check which experiments are DONE vs still running
4. **Log files**: `ssh server "tail -100 /path/to/training.log"` if no other source
5. **docs/research_contract.md**: intended claims and experiment design

Assemble the key information:
- What experiments were run (method, dataset, config)
- Main metrics and baseline comparisons (deltas)
- The intended claim these experiments were designed to test
- Any known confounds or caveats

### Step 1.5: Deterministic evidence pre-check (before spending a Codex call)

For every claim that cites a specific number + a source file, verify the evidence
*exists* mechanically — no model call — to catch **hallucinated evidence** before
the jury runs (see [`shared-references/evidence-precheck.md`](../shared-references/evidence-precheck.md)).

**1. Build the claims list.** From the cited numbers and their result files, write
`[{"id", "value", "source"}, ...]` to `.aris/claims.json` (`source` is the result
file/glob relative to the project root; `value` is the cited number or string).

**2. Run the pre-check — this is a real step, not a suggestion.** Execute the block
below (resolver per integration-contract §2, **Policy B**: warn-and-skip if the helper
is unresolved — never block the audit):

```bash
# Policy B = warn-and-skip: nothing here may abort the audit. cd is non-fatal, the
# helper run is explicitly non-blocking, no pipefail-fragile pipe.
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)" 2>/dev/null || true
if [ -z "${ARIS_REPO:-}" ] && [ -f .aris/installed-skills.txt ]; then
    ARIS_REPO=$(awk -F'\t' '$1=="repo_root"{print $2; exit}' .aris/installed-skills.txt 2>/dev/null) || true
fi
EVIDENCE_CHECK=".aris/tools/evidence_check.py"
[ -f "$EVIDENCE_CHECK" ] || EVIDENCE_CHECK="tools/evidence_check.py"
[ -f "$EVIDENCE_CHECK" ] || { [ -n "${ARIS_REPO:-}" ] && EVIDENCE_CHECK="$ARIS_REPO/tools/evidence_check.py"; }
[ -f "$EVIDENCE_CHECK" ] || EVIDENCE_CHECK=""

mkdir -p .aris
if [ -n "$EVIDENCE_CHECK" ]; then
    # NB: evidence_check exits 1 when it FINDS hallucinated evidence (value_not_found /
    # path_missing) — that is the useful signal, NOT a failure. So judge success by
    # whether valid JSON was produced, never by exit code. `|| true` keeps set -e calm.
    python3 "$EVIDENCE_CHECK" . --batch .aris/claims.json > .aris/evidence_precheck.json 2>.aris/evidence_precheck.err || true
    if [ -s .aris/evidence_precheck.json ] && python3 -c "import json,sys;json.load(open('.aris/evidence_precheck.json'))" 2>/dev/null; then
        cat .aris/evidence_precheck.json
    else
        echo "WARN: evidence_check produced no valid output (see .aris/evidence_precheck.err);" >&2
        echo "      pre-check skipped (Policy B); the Codex jury still runs." >&2
    fi
else
    echo "WARN: evidence_check.py not resolved at .aris/tools/, tools/, or \$ARIS_REPO/tools/." >&2
    echo "      Pre-check skipped (Policy B); the Codex jury still runs. Fix: rerun" >&2
    echo "      bash tools/install_aris.sh, export ARIS_REPO, or copy the helper to tools/." >&2
fi
```

The output is `{"results": [{id, value, source, status, ...}], "summary": {status: n}}`
with `status ∈ {verified, value_not_found, path_missing, unparseable}`.

**3. Act on the statuses.** Any claim returned `value_not_found` or `path_missing` is
**hallucinated evidence** — mark it `claim_supported: no` with
`integrity_status: evidence_not_found` immediately; do NOT spend a Codex call defending a
number that isn't in the data. `unparseable` claims (no usable value/source) just go to
the jury normally.

**4. Carry the per-claim status into Step 2.** Feed a small
`evidence pre-check: <id> → verified | value_not_found | path_missing | unparseable`
table (from `.aris/evidence_precheck.json`) into the Step-2 Codex prompt so the jury knows
which claims have real evidence to read. If the pre-check was skipped (helper unresolved),
say so in that slot rather than omitting it.

`verified` here means only that the cited evidence **exists** — whether it
**supports** the claim is still the Codex jury's call in Step 2 (a deterministic
gate DRIVES, it does not ACQUIT).

### Step 2: Codex Judgment

Send the collected results to Codex for objective evaluation:

```
mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    RESULT-TO-CLAIM EVALUATION

    I need you to judge whether experimental results support the intended claim.

    Intended claim: [the claim these experiments test]

    Experiments run:
    [list experiments with method, dataset, metrics]

    Results:
    [paste key numbers, comparison deltas, significance]

    Evidence pre-check (deterministic, from Step 1.5):
    [per-claim: <id> → verified | value_not_found | path_missing.
     A value_not_found/path_missing means the cited number is NOT in its result
     file — treat that claim as having no evidence; do not defend it. `verified`
     means the number exists in the file — YOU still judge whether it supports
     the claim.]

    Baselines:
    [baseline numbers and sources — reproduced or from paper]

    Known caveats:
    [any confounding factors, limited datasets, missing comparisons]

    Please evaluate:
    1. claim_supported: yes | partial | no
    2. what_results_support: what the data actually shows
    3. what_results_dont_support: where the data falls short of the claim
    4. missing_evidence: specific evidence gaps
    5. suggested_claim_revision: if the claim should be strengthened, weakened, or reframed
    6. next_experiments_needed: specific experiments to fill gaps (if any)
    7. confidence: high | medium | low

    Be honest. Do not inflate claims beyond what the data supports.
    A single positive result on one dataset does not support a general claim.
```

### Step 3: Parse and Normalize

Extract structured fields from Codex response:

```markdown
- claim_supported: yes | partial | no
- what_results_support: "..."
- what_results_dont_support: "..."
- missing_evidence: "..."
- suggested_claim_revision: "..."
- next_experiments_needed: "..."
- confidence: high | medium | low
```

### Step 3.5: Check Experiment Integrity (if audit exists)

**Skip this step if `EXPERIMENT_AUDIT.json` does not exist.**

```
if EXPERIMENT_AUDIT.json exists:
    read integrity_status from file
    attach to verdict output:
        integrity_status: pass | warn | fail

    if integrity_status == "fail":
        append to verdict: "[INTEGRITY CONCERN] — audit found issues, see EXPERIMENT_AUDIT.md"
        downgrade confidence to "low" regardless of Codex judgment

    if integrity_status == "warn":
        append to verdict: "[INTEGRITY: WARN] — audit flagged potential issues"
else:
    integrity_st