evolutionary-metric-ranking

What this skill does

# Evolutionary Metric Ranking

Methodology for systematically zooming into high-quality configurations across multiple evaluation metrics using per-metric percentile cutoffs, intersection-based filtering, and evolutionary optimization. Domain-agnostic principles with quantitative trading case studies.

**Companion skills**: `rangebar-eval-metrics` (metric definitions) | `adaptive-wfo-epoch` (WFO integration) | `backtesting-py-oracle` (SQL validation)

---

> **Self-Evolving Skill**: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

## When to Use This Skill

Use this skill when:

- Ranking and filtering configs/strategies/models across multiple quality metrics
- Searching for optimal per-metric thresholds that select the best subset
- Identifying which metrics are binding constraints vs inert dimensions
- Running multi-objective optimization (Optuna TPE / NSGA-II) over filter parameters
- Performing forensic analysis on optimization results (universal champions, feature themes)
- Designing a metric registry for pluggable evaluation systems

---

## Core Principles

### P1 - Percentile Ranks, Not Raw Values

Raw metric values live on incompatible scales (Kelly in [-1,1], trade count in [50, 5000], Omega in [0.8, 2.0]). Percentile ranking normalizes every metric to [0, 100], making cross-metric comparison meaningful.

```
Rule: scipy.stats.rankdata(method='average') scaled to [0, 100]
      None/NaN/Inf -> percentile 0 (worst)
      "Lower is better" metrics -> negate before ranking (100 = best)
```

**Why average ties**: Tied values receive the mean of the ranks they would span. This prevents artificial discrimination between genuinely identical values.

### P2 - Independent Per-Metric Cutoffs

Each metric gets its own independently-tunable cutoff. `cutoff=20` means "only configs in the top 20% survive this filter." This creates a 12-dimensional (or N-dimensional) search space where each axis controls one quality dimension.

```
cutoff=100 -> no filter (everything passes)
cutoff=50  -> top 50% survives
cutoff=10  -> top 10% survives (stringent)
cutoff=0   -> nothing passes
```

**Why independent, not uniform**: Different metrics have different discrimination power. Uniform tightening (all metrics at the same cutoff) wastes filtering budget on inert dimensions while under-filtering on binding constraints.

### P3 - Intersection = Multi-Metric Excellence

A config survives the final filter only if it passes **ALL** per-metric cutoffs simultaneously. This intersection logic ensures no single-metric champion sneaks through with terrible performance elsewhere.

```
survivors = metric_1_pass AND metric_2_pass AND ... AND metric_N_pass
```

**Why intersection, not scoring**: Weighted-sum scoring hides metric failures. A config with 99th percentile Sharpe but 1st percentile regularity would score well in a weighted sum but is clearly deficient. Intersection enforces minimum quality across every dimension.

### P4 - Start Wide Open, Tighten Evolutionarily

All cutoffs default to 100% (no filter). The optimizer progressively tightens cutoffs to find the combination that best satisfies the chosen objective. This is the opposite of starting strict and relaxing.

```
Initial state:  All cutoffs = 100 (1008 configs survive)
After search:   Each cutoff independently tuned (11 configs survive)
```

**Why start wide**: Starting strict risks missing the global optimum by immediately excluding configs that would survive under a different cutoff combination. Wide-to-narrow exploration is characteristic of global optimization.

### P5 - Multiple Objectives Reveal Different Truths

No single objective function captures "quality." Run multiple objectives and compare survivor sets. Configs that survive **all** objectives are the most robust.

| Objective                | Asks                                  | Reveals                                       |
| ------------------------ | ------------------------------------- | --------------------------------------------- |
| max_survivors_min_cutoff | Most configs at tightest cutoffs?     | Efficient frontier of quantity vs stringency  |
| quality_at_target_n      | Best quality in top N?                | Optimal cutoffs for a target portfolio size   |
| tightest_nonempty        | Absolute tightest with >= 1 survivor? | Universal champion (sole survivor)            |
| pareto_efficiency        | Survivors vs tightness trade-off?     | Full Pareto front (NSGA-II)                   |
| diversity_reward         | Are cutoffs non-redundant?            | Which metrics provide independent information |

**Cross-objective consistency**: A config that appears in ALL objective survivor sets is the most defensible selection. One that appears in only one is likely an artifact of that objective's bias.

### P6 - Binding Metrics Identification

After optimization, identify **binding metrics** - those that would increase the intersection if relaxed to 100%. Non-binding metrics are either already loose or perfectly correlated with a binding metric.

```
For each metric with cutoff < 100:
    Relax this metric to 100, keep others fixed
    If intersection grows: this metric IS binding
    If intersection unchanged: this metric is redundant at current cutoffs
```

**Why this matters**: Binding metrics are the actual constraints on your quality frontier. Effort to improve configs should focus on binding dimensions.

### P7 - Inert Dimension Detection

A metric is **inert** if it provides zero discrimination across the population. Detect this before optimization to reduce dimensionality.

```
If max(metric) == min(metric) across all configs: INERT
If percentile spread < 5 points: NEAR-INERT
```

**Action**: Remove inert metrics from the search space or permanently set their cutoff to 100. Including them wastes optimization budget.

### P8 - Forensic Post-Analysis

After optimization, perform forensic analysis to extract actionable insights:

1. **Universal champions** - configs surviving ALL objectives
2. **Feature frequency** - which features appear most in survivors
3. **Metric binding sequence** - order in which metrics become binding as cutoffs tighten
4. **Tightening curve** - intersection size vs uniform cutoff (100% -> 5%)
5. **Metric discrimination power** - which metric kills the most configs at each tightening step

---

## Architecture Pattern

```
Metric JSONL files (pre-computed)
        |
        v
MetricSpec Registry  <-- Defines name, direction, source, cutoff var
        |
        v
Percentile Ranker    <-- scipy.stats.rankdata, None->0, flip lower-is-better
        |
        v
Per-Metric Cutoff    <-- Each metric independently filtered
        |
        v
Intersection         <-- Configs passing ALL cutoffs
        |
        v
Evolutionary Search  <-- Optuna TPE/NSGA-II tunes cutoffs
        |
        v
Forensic Analysis    <-- Cross-objective consistency, binding metrics
```

### MetricSpec Registry

The registry is the single source of truth for metric definitions. Each entry is a frozen dataclass:

```python
@dataclass(frozen=True)
class MetricSpec:
    name: str              # Internal key (e.g., "tamrs")
    label: str             # Display label (e.g., "TAMRS")
    higher_is_better: bool # Direction for percentile ranking
    default_cutoff: int    # Default percentile cutoff (100 = no filter)
    source_file: str       # JSONL filename containing raw values
    source_field: str      # Field name in JSONL records
```

**Design principle**: Adding a new metric = adding one MetricSpec entry. No other code changes required. The ranking, cutoff, intersection, and optimization machinery is fully generic.

### Env Var Convention

Each metric's cutoff is controlled by a namespaced environment variable:

```
RBP_RANK_CUT_{METRIC_NAME_UPPER} = integer [0, 100]
```

This enables:

- She