benchmark-framework

What this skill does

# Cognitive Skills Benchmarking Framework

## Overview

A rigorous framework for A/B/C testing reasoning patterns to empirically determine which cognitive methodologies perform best across problem categories. This framework enables data-driven pattern selection rather than heuristic-based choices.

## Why Benchmark?

Different reasoning patterns (ToT, BoT, SRC, HE, AR, DR, AT, RTR, NDF) claim different strengths, but without empirical measurement:
- We cannot validate these claims
- We cannot quantify trade-offs (quality vs. cost vs. time)
- We cannot track improvement over time
- Pattern selection remains subjective

This framework provides scientific rigor to cognitive skill evaluation.

---

## Benchmark Structure

### Problem Set Organization

```
benchmark-problems/
├── optimization/           # ToT territory
│   ├── easy/              # 5-10 min problems
│   ├── medium/            # 15-30 min problems
│   └── hard/              # 30-60 min problems
├── exploration/           # BoT territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── diagnosis/             # HE territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── security/              # AR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── tradeoffs/             # DR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── novel/                 # AT territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── time-critical/         # RTR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
└── stakeholder/           # NDF territory
    ├── easy/
    ├── medium/
    └── hard/
```

### Problem Definition Schema

```yaml
# problem-template.yaml
problem_id: "OPT-001"
domain: "optimization"
difficulty: "medium"
title: "API Rate Limiter Design"
description: |
  Design a rate limiting system for a public API that handles
  10,000 requests/second with fair distribution across users.

context:
  constraints:
    - "Must handle burst traffic gracefully"
    - "Sub-millisecond latency requirement"
    - "Distributed deployment across 5 regions"
  resources:
    - "Redis cluster available"
    - "Current architecture uses nginx"

evaluation_criteria:
  - criterion: "Scalability"
    weight: 0.3
    rubric: |
      5: Handles 10x traffic with linear cost
      4: Handles 5x traffic efficiently
      3: Handles 2x traffic
      2: Handles current load only
      1: Cannot meet requirements

  - criterion: "Fairness"
    weight: 0.25
    rubric: |
      5: Per-user fairness with adaptive limits
      4: Per-user fairness with fixed limits
      3: Global fairness only
      2: Basic fairness, exploitable
      1: No fairness consideration

  - criterion: "Implementability"
    weight: 0.25
    rubric: |
      5: Clear implementation path, <1 week
      4: Implementation path, 1-2 weeks
      3: Requires some research, 2-4 weeks
      2: Significant unknowns
      1: Impractical to implement

  - criterion: "Operational Simplicity"
    weight: 0.2
    rubric: |
      5: Self-healing, minimal ops burden
      4: Standard monitoring/alerting sufficient
      3: Requires dedicated monitoring
      2: High operational complexity
      1: Operational nightmare

ground_truth:
  known_good_solutions:
    - "Token bucket with Redis MULTI/EXEC"
    - "Sliding window log with sorted sets"
  common_pitfalls:
    - "Race conditions in distributed counting"
    - "Memory explosion with naive approaches"
  expert_rating: 4.2  # If available

tags:
  - "distributed-systems"
  - "performance"
  - "redis"
```

---

## Metrics Framework

### Core Metrics

| Metric | Type | Range | Description |
|--------|------|-------|-------------|
| **Quality Score** | Aggregate | 0-100 | Weighted sum of evaluation criteria |
| **Confidence** | Self-reported | 0-100% | Pattern's reported confidence in solution |
| **Token Cost** | Integer | 0-∞ | Total tokens consumed (input + output) |
| **Execution Time** | Duration | ms | Wall-clock time to solution |
| **Correctness** | Binary/Partial | 0-1 | Does solution actually work? |
| **Completeness** | Percentage | 0-100% | How much of the problem addressed? |
| **Human Preference** | Rank | 1-N | Human ranking among alternatives |

### Quality Score Calculation

```python
def calculate_quality_score(solution, criteria):
    """
    Calculate weighted quality score from rubric evaluations.

    Args:
        solution: The solution being evaluated
        criteria: List of (criterion, weight, score) tuples

    Returns:
        Quality score 0-100
    """
    weighted_sum = 0
    total_weight = 0

    for criterion, weight, score in criteria:
        # Score is 1-5, normalize to 0-20, then weight
        normalized = (score - 1) * 25  # 1->0, 5->100
        weighted_sum += normalized * weight
        total_weight += weight

    return weighted_sum / total_weight if total_weight > 0 else 0
```

### Efficiency Metrics

```python
@dataclass
class EfficiencyMetrics:
    tokens_per_quality_point: float  # Lower is better
    time_per_quality_point: float    # Lower is better
    quality_per_minute: float        # Higher is better

    @classmethod
    def calculate(cls, quality: float, tokens: int, time_ms: int):
        return cls(
            tokens_per_quality_point=tokens / max(quality, 1),
            time_per_quality_point=time_ms / max(quality, 1),
            quality_per_minute=(quality * 60000) / max(time_ms, 1)
        )
```

### Confidence Calibration

Track how well self-reported confidence predicts actual quality:

```python
def calibration_score(predictions: List[Tuple[float, float]]) -> float:
    """
    Calculate calibration: does confidence predict quality?

    Args:
        predictions: List of (confidence, actual_quality) pairs

    Returns:
        Calibration score (-1 to 1, 1 is perfect)
    """
    if len(predictions) < 10:
        return None  # Insufficient data

    # Bin by confidence and compare to actual
    bins = defaultdict(list)
    for conf, quality in predictions:
        bin_key = int(conf // 10) * 10  # 0-10, 10-20, etc.
        bins[bin_key].append(quality)

    errors = []
    for bin_center, qualities in bins.items():
        expected = bin_center + 5  # Center of bin
        actual = sum(qualities) / len(qualities)
        errors.append(abs(expected - actual))

    # Average error, inverted and normalized
    avg_error = sum(errors) / len(errors) if errors else 50
    return 1 - (avg_error / 50)  # 0 error -> 1, 50 error -> 0
```

---

## A/B/C Testing Protocol

### Experimental Design

```yaml
experiment:
  id: "EXP-2024-001"
  hypothesis: "ToT outperforms BoT on optimization problems"

  conditions:
    - name: "baseline"
      pattern: "direct_analysis"
      description: "No specialized pattern"

    - name: "condition_a"
      pattern: "tree_of_thoughts"
      description: "ToT with default parameters"

    - name: "condition_b"
      pattern: "breadth_of_thought"
      description: "BoT with default parameters"

    - name: "condition_c"
      pattern: "tree_of_thoughts"
      parameters:
        max_branches: 5
        pruning_threshold: 0.6
      description: "ToT with aggressive pruning"

  problem_set:
    domain: "optimization"
    difficulties: ["medium", "hard"]
    sample_size: 30  # Problems per condition

  randomization:
    seed: 42
    counterbalancing: true  # Vary problem order

  controls:
    temperature: 0.7  # Fixed across conditions
    max_tokens: 8000  # Fixed across conditions
    time_limit: 300000  # 5 minutes per problem
```

### Statistical Requirements

#### Sample Size Calculation

```python
def required_sample_size(
    effect_size: float = 0.5,  # Cohen's d
    alpha: float = 0.05,       # Significance level
    power: float = 0.80        # Statistical power
) -> int:
    """
    Calculate minimum sample size for meaningful comparison.

    For quality score comparisons (continuous 0-100):
    - Small effect (d=0.2): n=393 per condition
    - Medium effect (d=0.5): n=64 per condition
    - Large effect (