test-skills

What this skill does

# Test All Skills — Full Composition Validation
<!-- markdownlint-disable MD040 MD060 -->

Applies **every** skill into sandboxes in topological (dependency) order, validates each layer, fixes broken skill markdowns, and ports fixes back to `skills/`. This tests real-world skill composition — not just individual skills in isolation.

## Quick Start

```
1. Scan all skills/*/SKILL.md — parse frontmatter, build dependency graph
2. Topologically sort skills, group into sandbox families
3. Present execution plan to user for approval
4. Per sandbox group: scaffold → layered implementation loop → report
5. Final report at test-results/report.md
```

## How It Works

```
┌────────────────────────────────────────────────────────────────────┐
│                     TEST ALL SKILLS                                │
│                                                                    │
│  ┌──────────────┐    ┌──────────────┐    ┌─────────────────────┐  │
│  │ SCAN & PLAN  │───▶│  INIT GROUP  │───▶│ LAYERED IMPL LOOP   │  │
│  │ parse deps,  │    │  sandbox per │    │                     │  │
│  │ topo sort,   │    │  family      │    │  for skill in order:│  │
│  │ group skills │    └──────────────┘    │    tag ──▶ impl     │  │
│  └──────────────┘                        │    ──▶ validate     │  │
│                                          │    ──▶ on error:    │  │
│                                          │       fix markdown  │  │
│                                          │       revert to tag │  │
│                                          │       retry (max 3) │  │
│                                          │    ──▶ on success:  │  │
│                                          │       tag + next    │  │
│                                          └─────────────────────┘  │
│                                                    │               │
│                                          ┌─────────▼───────────┐  │
│                                          │      REPORT         │  │
│                                          │  test-results/      │  │
│                                          │  report.md          │  │
│                                          └─────────────────────┘  │
└────────────────────────────────────────────────────────────────────┘

Each skill is git-tagged so any skill can be reverted independently.
```

## Prerequisites

- `skills/` directory with polished skill markdowns
- `sandbox/` directory (working area — created per group)
- Node.js / Bun runtime for validation
- `scripts/validate.sh` from this skill's directory

## Workflow Steps

### Phase 0: Scan & Plan

Parse every saved skill, build the dependency graph, topologically sort, group into sandbox families, and present the execution plan.

**Agent instructions:**

1. **Scan all skills:**

```bash
ls skills/*/SKILL.md
```

1. **Parse frontmatter** from each skill:
   - Extract `name` field
   - Extract `dependencies` array (may be empty or missing)
   - Record skill path

2. **Build dependency graph** — see [DEPENDENCY_RESOLUTION.md](references/DEPENDENCY_RESOLUTION.md):
   - Create adjacency list: `skill → [dependencies]`
   - Detect cycles (error if found — report and stop)
   - Topologically sort using Kahn's algorithm
   - Group into sandbox families using connected components

3. **Known sandbox groups** (based on current skills):

**Group A — Next.js Family** (~38 skills connected through dependency chains):

Skills that directly or transitively depend on `create-next`, `env-config`, `docker`, or connect through shared dependencies like `auth`, `db`, `storage`:

```
Layer 0 (no deps):     create-next, docker, env-config
Layer 1:               add-shadcn, add-pwa, add-seo, auth, db, storage, email, ai-core
Layer 2:               auth-dev, storage-ui, media-bunny, realtime, image-editor, ai-chat,
                       payments, queue, ai-image-gen, ai-video-gen, ai-rag-ingest, ai-rag-viewer
Layer 3:               cms, ai-tools, ai-reasoning, ai-rag-vectors,
                       embeddable-widget, voice-retell
Layer 4:               ai-memory, ai-tasks, ai-artifacts, ai-generative-ui, ai-rag-chat, ai-mcp,
                       knowledge-sync
Layer 5:               ai-rag-app
```

**Group B — Standalone** (skills with no deps that don't assume Next.js):

```
setup-lefthook, mcp-server, yt-dlp, lottie, react-flow, react-three-fiber, e2e,
env-from-1password, workflow
```

**Note:** `e2e` has no `dependencies` frontmatter but assumes a Next.js app exists. If it fails standalone, move it to Group A at the appropriate layer.

### Phase 0b: Catalog Health Validation (required)

Before presenting the plan, validate catalog consistency so renamed/deleted skills are caught early.

Run:

```bash
# Skills on disk
DISK_SKILLS=$(ls -d skills/*/SKILL.md 2>/dev/null | sed 's|skills/||;s|/SKILL.md||' | sort)

# Skills listed in README markdown tables
README_SKILLS=$(rg -o '(?<=\\| `)[^`]+' README.md | sort)

# Skills listed in add-feature catalog bullets
ADD_FEATURE_SKILLS=$(rg -o '^- [a-z0-9-]+' skills/add-feature/SKILL.md | sed 's/^- //' | sort)

# Skill names referenced by scaffold DAG
DAG_SKILLS=$(jq -r '
  [
    .tiers[].layers[]?.skills[]?,
    .tiers[].packs[]?[]?.skills[]?,
    .extensionPoints[]?.id?
  ] | .[]
' skills/add-project/references/scaffold-dag.json | sort -u)

echo "---- On disk but missing from README ----"
comm -23 <(echo "$DISK_SKILLS") <(echo "$README_SKILLS")

echo "---- In README but missing on disk ----"
comm -13 <(echo "$DISK_SKILLS") <(echo "$README_SKILLS")

echo "---- In add-feature but missing on disk ----"
comm -13 <(echo "$DISK_SKILLS") <(echo "$ADD_FEATURE_SKILLS")

echo "---- On disk but missing from add-feature ----"
comm -23 <(echo "$DISK_SKILLS") <(echo "$ADD_FEATURE_SKILLS")

echo "---- In scaffold DAG but missing on disk ----"
comm -13 <(echo "$DISK_SKILLS") <(echo "$DAG_SKILLS")
```

If any of these lists are non-empty, treat as a catalog drift defect and fix docs before continuing skill composition tests.

1. **Present plan to user** — show:
   - Total skill count per group
   - Layered execution order for Group A
   - List of standalone skills for Group B
   - Estimated scope ("Group A: ~38 skills across 6 layers, Group B: ~8 standalone")

2. **Wait for user approval** before proceeding. User may:
   - Approve full plan
   - Request only Group A or Group B
   - Request a subset of layers
   - Skip specific skills

---

### Phase 1: Initialize Sandbox (per group)

**Agent instructions — repeat for each sandbox group:**

1. **Determine sandbox directory:**

```bash
# Group A uses: sandbox/
# Group B uses: sandbox-standalone/
```

1. **Clean and scaffold:**

```bash
# Group A (Next.js):
rm -rf sandbox/* sandbox/.* 2>/dev/null
cd sandbox && bunx create-next-app@latest . --typescript --tailwind --eslint --app --src-dir --no-import-alias --use-bun && cd ..

# Group B (Node.js / varies per skill):
rm -rf sandbox-standalone/* sandbox-standalone/.* 2>/dev/null
cd sandbox-standalone && bun init -y && cd ..
```

1. **Git init + baseline commit:**

```bash
cd sandbox
git init
git add -A
git commit -m "baseline: scaffold for test-skills group-a"
cd ..
```

1. **Verify clean build:**

```bash
cd sandbox && bun run build && cd ..
```

---

### Phase 2: Layered Implementation Loop

Process skills in topological order within each group. Each skill builds on top of the previous ones.

**Set tracking variables:**

```
RESULTS = {}          # skill → { status, attempts, fixes }
CURRENT_LAYER = 0
```

**For each skill in topological order:**

#### Step 2a: Tag Before Implementation

Create a git tag so this skill can be reverted independently:

```bash
cd sandbox
git tag "before-<SKILL_NAME>"
cd ..
```

#### Step 2b: Implement

Read the skill markdown and implement it on top of the current sandbox state.

**Agent instructions:**

1. Read `skills/<SKILL_NAME>/SKILL.md` completely
2. Follow every Setup Step / Implementation instruction
3. Create all files from "Wh