run-simulations

What this skill does

# Run Simulations Skill

You are helping set up and run trading engine simulations, then performing comprehensive diagnostic analysis.

## Current Phase: DUAL-FOCUS (Validation + Optimization)

**System confidence: ~85% accurate.** The simulation system is largely trustworthy, but discrepancies still exist and are critical to find.

### Priority #1: ACCURACY DISCREPANCIES (Most Important)
Any gap between expected vs actual behavior is a potential bug or misunderstanding. These must be surfaced prominently:
- Does behavior match config parameters?
- Are there logical inconsistencies in the data?
- Do results align with what the strategy SHOULD produce?

**When you spot a discrepancy, flag it prominently.** Even small accuracy issues compound over thousands of trades.

### Priority #2: PROFIT OPTIMIZATION (Active Research)
With a trustworthy system, we're now actively researching:
- Which configs perform best?
- What market conditions favor which strategies?
- How do parameters affect PnL?

Surface interesting profit patterns AND accuracy concerns. Both matter now.

---

**IMPORTANT**: This skill should be invoked BEFORE running simulations to help with setup decisions (config selection, segment selection via collections, batch sizing).

## Recommended Batch Size

**Current recommendation: 2,000-3,000 simulations per batch.**
- Runs quickly (~30-60 seconds with tick caching)
- Provides statistically meaningful results
- More than 3,000 doesn't add much value at current stage
- Formula: configs × segments = total runs (e.g., 20 configs × 100 segments = 2,000 runs)

## Phase 1: Setup & Run

### ⚠️ USER INPUT TAKES PRIORITY

**If the user provides ANY arguments, instructions, or context when invoking this skill, those take absolute priority over all defaults below.**

- User says "run config X" → run config X, ignore default config selection
- User says "test these 5 segments" → test those 5 segments, ignore default segment query
- User asks a specific question → answer that question, don't run the full default workflow
- User provides partial instructions → fill in gaps with defaults, but honor what they specified

**The defaults below are ONLY for when the skill is invoked with no arguments at all.**

---

### Default Behavior (no args only)
Pick a diverse spread of configs and segments using **segment collections** (legacy filters are deprecated):
1. Query available configs: `SELECT name, config FROM strategy_configs ORDER BY name`
2. Select 10-20 configs covering different indicator types (EMA, RSI, Bollinger, combined)
3. Choose or create a segment collection (AND-only annotations in v1):
   ```bash
   bun src/systems/trading-engine/bd-segments.ts list
   bun src/systems/trading-engine/bd-segments.ts create --name=high-activity --annotations=high_activity --limit=100
   bun src/systems/trading-engine/bd-segments.ts preview --name=high-activity
   ```
4. Run batch simulation:
   ```bash
   bun src/systems/trading-engine/batch-runner.ts --config-pattern="<pattern>" --collection=high-activity --save
   ```

### Batch Runner CLI Reference

**Location:** `src/systems/trading-engine/batch-runner.ts`

**Usage:**
```bash
bun src/systems/trading-engine/batch-runner.ts [options]
```

**Options:**
| Option | Short | Description |
|--------|-------|-------------|
| `--config=ID` | `-c` | Strategy config ID (can specify multiple) |
| `--all-configs` | `-a` | Run all available strategy configs |
| `--config-pattern=PAT` | `-p` | Run configs matching name pattern (SQL LIKE) |
| `--strategy-type=TYPE` | | Filter configs by strategy type (volatility, arb, grid) |
| `--collection=NAME|ID` | `-C` | Segment collection to run (required) |
| `--show-selection` | | Preview collection selection and exit |
| `--capital=CENTS` | | Initial capital in cents (default: 10000 = $100) |
| `--workers=N` | `-w` | Number of parallel workers (default: 8) |
| `--save` | `-S` | Save results to database |
| `--dry-run` | | Show what would run without executing |
| `--warmup-candles=N` | | Pre-load N candles before segment for indicator warmup (default: 50) |
| `--db-pool-max=N` | | DB pool size for main batch process (default: 10) |
| `--db-pool-max-worker=N` | | DB pool size for worker processes (default: 3) |

**Examples:**
```bash
# Run one config on a segment collection
bun src/systems/trading-engine/bd-segments.ts create --name=swings100 --annotations=high_activity
bun src/systems/trading-engine/batch-runner.ts --config=abc123 --collection=swings100 --save

# Run all configs on a collection
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=swings50 --save

# Run all configs on specific segment IDs (snapshot collection)
bun src/systems/trading-engine/bd-segments.ts create --name=segment-set --segment-ids=949,950,951,952 --mode=snapshot
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=segment-set --save

# Run configs matching "RSI-*" on high_activity segments
bun src/systems/trading-engine/bd-segments.ts create --name=rsi-activity --annotations=high_activity
bun src/systems/trading-engine/batch-runner.ts --config-pattern="RSI-%" --collection=rsi-activity --save

# Dry run to see what would execute
bun src/systems/trading-engine/batch-runner.ts --all-configs --collection=swings100 --dry-run
```

### Key CLI Flags (Quick Reference)
- `--collection=NAME|ID` or `-C`: REQUIRED segment collection selector
- `--show-selection`: Preview resolved segment list and exit
- `--config-pattern="TEST-%"` or `-p`: Filter configs by name pattern
- `--strategy-type=grid`: Filter configs by strategy type

## Phase 2: Comprehensive Analysis (ALWAYS RUN)

After simulations complete, run ALL of the following diagnostic queries. Present results in tables.

### 2.1 Core Stats
```sql
SELECT
  sc.name,
  COUNT(*) as runs,
  ROUND(AVG(trade_count)::numeric, 1) as avg_trades,
  ROUND(STDDEV(trade_count)::numeric, 1) as stddev_trades,
  MIN(trade_count) as min_trades,
  MAX(trade_count) as max_trades
FROM (
  SELECT r.id, sc.name, COUNT(t.id) as trade_count
  FROM sim_runs r
  JOIN strategy_configs sc ON r.config_id = sc.id
  LEFT JOIN sim_trades t ON t.run_id = r.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
  GROUP BY r.id, sc.name
) sub
JOIN strategy_configs sc ON sc.name = sub.name
GROUP BY sc.name
ORDER BY avg_trades;
```

### 2.2 Entry Signal Frequency (RED FLAG: median < 60s is suspicious)
```sql
WITH entries AS (
  SELECT
    t.run_id, sc.name as config, t.entry_at,
    LAG(t.entry_at) OVER (PARTITION BY t.run_id ORDER BY t.entry_at) as prev_entry
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
)
SELECT
  config,
  COUNT(*) as entry_gaps,
  ROUND(AVG((entry_at - prev_entry)/1000.0)::numeric, 1) as avg_gap_sec,
  ROUND(PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY (entry_at - prev_entry)/1000.0)::numeric, 1) as median_gap_sec,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 1000) as gaps_under_1s,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 5000) as gaps_under_5s,
  COUNT(*) FILTER (WHERE entry_at - prev_entry < 60000) as gaps_under_1min
FROM entries
WHERE prev_entry IS NOT NULL
GROUP BY config
ORDER BY median_gap_sec;
```
**Interpretation**: Median gaps < 60s suggest overtrading. Healthy configs should have gaps in minutes (20-30min typical). Gaps < 1s are a MAJOR RED FLAG - indicates indicator firing constantly.

### 2.3 Position Overlap
```sql
-- How many concurrent positions on average?
WITH trade_events AS (
  SELECT t.run_id, sc.name as config, t.entry_at as ts, 1 as delta
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW() - INTERVAL '1 hour'
  UNION ALL
  SELECT t.run_id, sc.name as config, t.exit_at as ts, -1 as delta
  FROM sim_trades t
  JOIN sim_runs r ON t.run_id = r.id
  JOIN strategy_configs sc ON r.config_id = sc.id
  WHERE r.created_at > NOW