skillify

What this skill does

# Skillify — The Meta Skill

> **Relationship to `/cross-modal-review`:** That skill is the manual mid-flow
> "second opinion" gate (one model reviews work product before commit). This
> skill's Phase 3 below uses `gbrain eval cross-modal` instead — three
> different-provider frontier models score-and-iterate on a documented
> dimension list *before* tests cement behavior. Use `/cross-modal-review`
> for ad-hoc second opinions; use Phase 3 here when skillifying a feature.

## Contract

A feature is "properly skilled" when all 11 checklist items pass. Item 3
(cross-modal eval) is informational in v1.1.0 — it does not gate the
skillpack-check audit, but a missing or stale receipt is surfaced so the
user knows where the gate stands.

## The Checklist

```
□ 1.  SKILL.md           — skill file with frontmatter + contract + phases
□ 2.  Code               — deterministic script if applicable
□ 3.  Cross-modal eval   — 3 frontier models from 3 providers; informational
□ 4.  Unit tests         — cover every branch of deterministic logic
□ 5.  Integration tests  — exercise live endpoints
□ 6.  LLM evals          — quality/correctness cases for LLM-involving steps
□ 7.  Resolver trigger   — entry in skills/RESOLVER.md with real user trigger phrases
□ 8.  Resolver eval      — test that triggers route to this skill
□ 9.  Check-resolvable   — DRY + MECE audit, no orphans
□ 10. E2E test           — smoke test: trigger → side effect
□ 11. Brain filing       — if it writes pages, entry in brain/RESOLVER.md
```

## Phase 0: Should This Be a Skill?

Before skillifying, check:
- Will this be invoked 2+ times? (One-off work ≠ skill)
- Is there >20 lines of logic? (Trivial helpers don't need full infrastructure)
- Does it have a clear trigger phrase a user would actually say?

If no to all three, it's a script, not a skill. Move on.

## Phase 1: Audit

```
Feature: [name]
Code: [path]
Missing items: [check each of the 11]
```

## Phase 2: Write SKILL.md + Code (items 1-2)

### SKILL.md frontmatter template (copy-paste):

```yaml
---
name: my-skill
version: 1.0.0
description: |
  One paragraph. What it does, when to use it.
triggers:
  - "trigger phrase users actually say"
  - "another real trigger"
tools:
  - exec
  - read
  - write
mutating: false  # true if it writes to brain/disk
---
```

Body must include: **Contract** (what it guarantees), **Phases** (step-by-step), **Output Format** (what it produces).

Extract deterministic code into `scripts/*.ts`.

## Phase 3: Cross-Modal Eval (item 3) — THE QUALITY GATE

### Why this comes before tests

Tests lock in behavior. If the behavior is mediocre, tests lock in mediocrity.
Cross-modal eval proves the quality bar FIRST, then tests cement it.

### Step 1: Pick a representative input

Choose the input that exercises the skill's hardest documented use case. If
unsure: use the primary trigger example from SKILL.md, or the most complex
real-world input from the last 7 days of memory files.

### Step 2: Run the skill, capture output

Run the skill on the representative input. The OUTPUT FILE is what gets
evaluated.

### Step 3: Run the eval gate

```bash
gbrain eval cross-modal \
  --task "What this skill is supposed to accomplish" \
  --output skills/<slug>/SKILL.md
```

The command runs 3 frontier models from 3 different providers in parallel,
scores the OUTPUT against the TASK on 5 documented dimensions, and writes a
receipt under `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` (the
sha-8 binds the receipt to the current SKILL.md content — re-running after
edits writes a new receipt).

**Default models** (override per slot via `--slot-a-model`, `--slot-b-model`,
`--slot-c-model`):

| Slot | Default | Provider |
|------|---------|----------|
| A | `openai:gpt-4o` | OpenAI |
| B | `anthropic:claude-opus-4-7` | Anthropic |
| C | `google:gemini-1.5-pro` | Google |

**These MUST be frontier models from DIFFERENT providers.** Using a single
provider's family or budget models defeats the purpose — different families
have less correlated blind spots. Refresh the list when a new model
generation ships.

**Pass criteria (BOTH must be true):**

1. Every dimension's mean across successful models ≥ 7.
2. No single model scored any dimension < 5 (the floor).

**Inconclusive:** fewer than 2 of 3 models returned parseable scores.
Receipt is still written (forensics) but the gate is not authoritative.
Exit code 2; CI wrappers should treat this as "did not run cleanly", not
"failed quality gate".

### Step 4: Cycle until you pass (≤3 cycles)

```
CYCLE 1:
  Eval → scores + top 10 improvements
  IF pass: → done, write tests
  ELSE:
    Apply top 10 improvements to the actual file
    Log: which improvements applied, what changed

CYCLE 2:
  Re-eval the FIXED output (same 3 models, same dimensions)
  Compare: before/after scores per dimension (track delta)
  IF pass: → done, write tests
  ELSE: apply remaining improvements + new ones

CYCLE 3 (final):
  Re-eval
  IF pass: → ship
  ELSE: → ship with KNOWN_GAPS section listing:
    - Which dimensions are still below 7
    - Which improvements couldn't be resolved
    - Why (e.g., "would require architectural change")
```

### Cycles + cost guardrails

- Default `--cycles 3` in TTY, `--cycles 1` in non-TTY (limits scripted
  bulk spend in CI loops).
- The command prints an estimated max-cost-per-cycle from a small pricing
  constant before each run. Real cost varies with prompt size; treat the
  estimate as a ceiling for default `--max-tokens 4000`.
- A `--budget-usd N` hard cap is a v0.27.x follow-up TODO.

### Provider configuration

Models resolve through the gbrain AI gateway. Configure once with:

```bash
gbrain providers test    # see what's configured
gbrain config            # set keys
```

Or set env vars: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
`GOOGLE_GENERATIVE_AI_API_KEY`, `TOGETHER_API_KEY`, etc. The gateway reads
from `~/.gbrain/config.json` plus `process.env`.

### Cost expectations

3 cycles × 3 models = 9 frontier calls max per run. With Opus-class +
GPT-4o-class + Gemini-1.5-Pro, expect $1–3 per full run on default
`--max-tokens 4000`. Receipts include the per-call model identifiers so
you can audit retroactively.

### Skip cross-modal eval when:

- Output is < 200 tokens (trivial — not worth 9 API calls).
- The skill is a thin wrapper around a single API call (one cycle is enough).

## Phase 4: Tests (items 4-6)

NOW that eval has proven quality, write tests that lock it in:

**Unit tests** — every branch of deterministic logic. Mock external calls.
**Integration tests** — hit real endpoints. Catch bugs mocks hide.
**LLM evals** — quality/correctness for LLM steps. Lighter than cross-modal eval — test specific behaviors.

## Phase 5: Resolver + Check-Resolvable (items 7-9)

1. Add to skills/RESOLVER.md with trigger phrases users ACTUALLY type
2. Resolver eval: feed triggers, assert correct routing
3. Check-resolvable:
   - Skill reachable from skills/RESOLVER.md (not orphaned)
   - No MECE overlap with other skills
   - No DRY violations (shared logic in lib/, not copy-pasted)
   - No ambiguous trigger routing

## Phase 6: E2E + Brain Filing (items 10-11)

- E2E smoke: full pipeline from trigger to side effect
- Brain filing: add to brain/RESOLVER.md if the skill writes brain pages

## Phase 7: Verify

```bash
bun test test/<skill>.test.ts                    # unit tests
gbrain skillify check skills/<slug>/scripts/<slug>.mjs --json | \
  jq '.[] | .items[] | select(.name | contains("Cross-modal"))'
ls ~/.gbrain/.gbrain/eval-receipts/              # receipt landed
gbrain check-resolvable --json | jq .ok          # resolver clean
```

## Worked Example: Skillifying a "summarize-pr" Feature

```
Phase 0: Yes — invoked weekly, 50+ lines, clear trigger "summarize this PR"
Phase 1: Audit → SKILL.md missing, no tests, no resolver entry. Score: 1/11
Phase 2: Write SKILL.md + extract script to scripts/summarize-pr.ts
Phase 3: Cross-modal eval cycle 1 →
  GPT-4o: goal=6, depth=5