multi-model-validation

What this skill does

# Multi-Model Validation

**Version:** 3.1.0
**Purpose:** Patterns for running multiple AI models in parallel via Claudish proxy with dynamic model discovery, session-based workspaces, and performance statistics
**Status:** Production Ready

## Overview

Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT-5, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves:

- **3-5x speedup** via parallel execution (15 minutes → 5 minutes)
- **Consensus-based prioritization** (issues flagged by all models are CRITICAL)
- **Diverse perspectives** (different models catch different issues)
- **Cost transparency** (know before you spend)
- **Free model discovery** (NEW v3.0) - find high-quality free models from trusted providers
- **Performance tracking** - identify slow/failing models for future exclusion
- **Data-driven recommendations** - optimize model shortlist based on historical performance

**Key Innovations:**

1. **Dynamic Model Discovery** (NEW v3.0) - Use `claudish --top-models` and `claudish --free` to get current available models with pricing
2. **Session-Based Workspaces** (NEW v3.0) - Each validation session gets a unique directory to prevent conflicts
3. **4-Message Pattern** - Ensures true parallel execution by using only Task tool calls in a single message
4. **Pattern 7-8** - Statistics collection and data-driven model recommendations

This skill is extracted from the `/review` command and generalized for use in any multi-model workflow.

---

## Related Skills

> **CRITICAL: Tracking Protocol Required**
>
> Before using any patterns in this skill, ensure you have completed the
> pre-launch setup from `orchestration:model-tracking-protocol`.
>
> Launching models without tracking setup = INCOMPLETE validation.

**Cross-References:**

- **orchestration:model-tracking-protocol** - MANDATORY tracking templates and protocols (NEW in v0.6.0)
  - Pre-launch checklist (8 required items)
  - Tracking table templates
  - Failure documentation format
  - Results presentation template
- **orchestration:quality-gates** - Approval gates and severity classification
- **orchestration:todowrite-orchestration** - Progress tracking during execution
- **orchestration:error-recovery** - Handling failures and retries

**Skill Integration:**

This skill (`multi-model-validation`) defines **execution patterns** (how to run models in parallel).
The `model-tracking-protocol` skill defines **tracking infrastructure** (how to collect and present results).

**Use both together:**
```yaml
skills: orchestration:multi-model-validation, orchestration:model-tracking-protocol
```

---

## Core Patterns

### Pattern 0: Session Setup and Model Discovery (NEW v3.0)

**Purpose:** Create isolated session workspace and discover available models dynamically.

**Why Session-Based Workspaces:**

Using a fixed directory like `ai-docs/reviews/` causes problems:
- ❌ Multiple sessions overwrite each other's files
- ❌ Stale data from previous sessions pollutes results
- ❌ Hard to track which files belong to which session

Instead, create a **unique session directory** for each validation:

```bash
# Generate unique session ID
SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="/tmp/${SESSION_ID}"

# Create session workspace
mkdir -p "$SESSION_DIR"

# Export for use by agents
export SESSION_ID SESSION_DIR

echo "Session: $SESSION_ID"
echo "Directory: $SESSION_DIR"

# Example output:
# Session: review-20251212-143052-a3f2
# Directory: /tmp/review-20251212-143052-a3f2
```

**Benefits:**
- ✅ Each session is isolated (no cross-contamination)
- ✅ Easy cleanup (`rm -rf $SESSION_DIR` when done)
- ✅ Session ID can be used for tracking in statistics
- ✅ Parallel sessions don't conflict

---

**Dynamic Model Discovery:**

**NEVER hardcode model lists.** Models change frequently - new ones appear, old ones deprecate, pricing updates. Instead, use `claudish` to get current available models:

```bash
# Get top paid models (best value for money)
claudish --top-models

# Example output:
#   google/gemini-3-pro-preview    Google     $7.00/1M   1048K   🔧 🧠 👁️
#   openai/gpt-5.1-codex           Openai     $5.63/1M   400K    🔧 🧠 👁️
#   x-ai/grok-code-fast-1          X-ai       $0.85/1M   256K    🔧 🧠
#   minimax/minimax-m2             Minimax    $0.64/1M   262K    🔧 🧠
#   z-ai/glm-4.6                   Z-ai       $1.07/1M   202K    🔧 🧠
#   qwen/qwen3-vl-235b-a22b-ins... Qwen       $0.70/1M   262K    🔧    👁️

# Get free models from trusted providers
claudish --free

# Example output:
#   google/gemini-2.0-flash-exp:free  Google     FREE      1049K   ✓ · ✓
#   mistralai/devstral-2512:free      Mistralai  FREE      262K    ✓ · ·
#   qwen/qwen3-coder:free             Qwen       FREE      262K    ✓ · ·
#   qwen/qwen3-235b-a22b:free         Qwen       FREE      131K    ✓ ✓ ·
#   openai/gpt-oss-120b:free          Openai     FREE      131K    ✓ ✓ ·
```

**Recommended Free Models for Code Review:**

| Model | Provider | Context | Capabilities | Why Good |
|-------|----------|---------|--------------|----------|
| `qwen/qwen3-coder:free` | Qwen | 262K | Tools ✓ | Coding-specialized, large context |
| `mistralai/devstral-2512:free` | Mistral | 262K | Tools ✓ | Dev-focused, excellent for code |
| `qwen/qwen3-235b-a22b:free` | Qwen | 131K | Tools ✓ Reasoning ✓ | Massive 235B model, reasoning |

**Model Selection Flow:**

```
1. Load Historical Performance (if exists)
   → Read ai-docs/llm-performance.json
   → Get avg speed, quality, success rate per model

2. Discover Available Models
   → Run: claudish --top-models (paid)
   → Run: claudish --free (free tier)

3. Merge with Historical Data
   → Add performance metrics to model list
   → Flag: "⚡ Fast", "🎯 High Quality", "⚠️ Slow", "❌ Unreliable"

4. Present to User (AskUserQuestion)
   → Show: Model | Provider | Price | Avg Speed | Quality
   → Suggest internal reviewer (ALWAYS)
   → Highlight top performers
   → Include 1-2 free models for comparison

5. User Selects Models
   → Minimum: 1 internal + 1 external
   → Recommended: 1 internal + 2-3 external
```

**Interactive Model Selection (AskUserQuestion with multiSelect):**

**CRITICAL:** Use AskUserQuestion tool with `multiSelect: true` to let users choose models interactively. This provides a better UX than just showing recommendations.

```typescript
// Use AskUserQuestion to let user select models
AskUserQuestion({
  questions: [{
    question: "Which external models should validate your code? (Internal Claude reviewer always included)",
    header: "Models",
    multiSelect: true,
    options: [
      // Top paid (from claudish --top-models + historical data)
      {
        label: "x-ai/grok-code-fast-1 ⚡",
        description: "$0.85/1M | Quality: 87% | Avg: 42s | Fast + accurate"
      },
      {
        label: "google/gemini-3-pro-preview",
        description: "$7.00/1M | Quality: 91% | Avg: 55s | High accuracy"
      },
      // Free models (from claudish --free)
      {
        label: "qwen/qwen3-coder:free 🆓",
        description: "FREE | Quality: 82% | 262K context | Coding-specialized"
      },
      {
        label: "mistralai/devstral-2512:free 🆓",
        description: "FREE | 262K context | Dev-focused, new model"
      }
    ]
  }]
})
```

**Remember Selection for Session:**

Store the user's model selection in the session directory so it persists throughout the validation:

```bash
# After user selects models, save to session
save_session_models() {
  local session_dir="$1"
  shift
  local models=("$@")

  # Always include internal reviewer
  echo "claude-embedded" > "$session_dir/selected-models.txt"

  # Add user-selected models
  for model in "${models[@]}"; do
    echo "$model" >> "$session_dir/selected-models.txt"
  done

  echo "Session models saved to $session_dir/selected-models.txt"
}

# Load session models for subsequent operations
load_