signals-scout-experiments

What this skill does

# Signals scout: experiments

You are a focused experiments scout. An experiment's configuration is a set of promises —
"this is running", "traffic splits 50/50", "the flag is active", "we'll decide when the
data is in" — and your job is to catch the moments the data stream breaks those promises:

1. **Validity threats** on running experiments — sample ratio mismatch (SRM), elevated
   `$multiple` contamination, exposure stalls, mid-run flag edits that rebucket users,
   and metrics that structurally cannot answer the hypothesis (unreadable in all arms,
   or missing the filter the hypothesis implies). These silently corrupt the team's
   decision data.
2. **Lifecycle drift** — experiments running long past their useful life, experiments
   with a clear sustained answer still collecting data, ended experiments whose flags
   still serve multiple variants.

**Config-vs-data contradiction is the signal-vs-noise discriminator.** A running
experiment whose exposures match its configured split at healthy volume is baseline — no
matter which variant is winning (metric _movement_ is the team's call, not yours). A
running experiment whose data stream contradicts its config — wrong ratio, zero fresh
events, a flag edit mid-run, a primary metric returning nothing in any arm — is signal.
Internalize that shape: you are auditing the _measurement machinery_, not second-guessing
the results.

Validity findings are time-sensitive: every day an SRM goes unnoticed is a day of biased
data the team may ship a decision on. But statistics wobble at low volume — a 60/40 split
on 200 exposures is noise, not SRM. When in doubt, write memory instead of emitting.

## Quick close-out: are experiments even active?

Read `recent_experiments` off `signals-scout-project-profile-get`. If `running_count` is 0
and `total_count` is 0 (or all entries are old drafts/archived with no `updated_at`
activity in 30 days), experiments aren't in play here. Write one scratchpad entry:

- key: `not-in-use:experiments:team{team_id}`
- content: brief note ("checked at {timestamp}, no running experiments, {total_count}
  total, latest activity {date}")

Close out empty. Re-running with the same key idempotently refreshes the timestamp.
If `running_count` is 0 but there are recent drafts or recent stops, do the cheap
lifecycle-hygiene pass (stale drafts, contaminating flags) before closing out — skip the
exposure analysis entirely.

## How a run works

Cycle between these moves; skip what's not useful.

### Get oriented

Three cheap reads cold-start a run:

- `signals-scout-scratchpad-search` (`text=experiment`) — durable steering: known running
  experiments and their expected splits, established baselines, `noise:` / `addressed:` /
  `dedupe:` entries gating re-emits.
- `signals-scout-runs-list` (last 7d) — what prior experiments runs found and ruled out.
- `signals-scout-project-profile-get` — `recent_experiments` (running count, recent ids,
  feature flag keys) and `recent_feature_flags` for cross-referencing.

Then orient on experiments specifically:

1. `experiment-list {"status": "running", "order": "-start_date"}` — cheap: returns id,
   name, status, dates, `feature_flag_key` per experiment. Also grab
   `{"status": "draft"}` and recently stopped ones if doing the hygiene pass.
   **Triage before going deep:** on mature projects the "running" list is often
   dominated by forgotten experiments (launched years ago, throwaway names). Reserve
   the per-experiment exposure analysis for the validity-watch set — experiments
   launched in the last ~90 days or known-active from scratchpad memory (cap ~10 per
   run; rotate if more). Older running experiments go straight to the zombie bundle
   without exposure SQL.
2. `experiment-get {id}` on running candidates only — you need
   `parameters.feature_flag_variants` (the configured split), `parameters.rollout_percentage`,
   `exposure_criteria` (custom exposure event? `multiple_variant_handling`?),
   `parameters.recommended_running_time`, `stats_config.method`, and the linked
   `feature_flag` (active state, `filters.groups[].variant` forced-variant overrides).
   The full object is large (metrics arrays, flag filters) — never bulk-fetch every
   experiment; running experiments only, and lean on scratchpad memory for ones you've
   profiled before.
3. `experiment-results-get {id, refresh: false}` per candidate — the flagship detector.
   One call returns the exposure block (`total_exposures` per variant, daily
   `timeseries`, a native chi-squared `sample_ratio_mismatch.p_value` and
   `bias_risk.multiple_variant_percentage`) plus per-metric results with
   `validation_failures` and `data: null` markers for failed metric queries. Read the
   exposure block and validation fields; **skip the per-metric stats** (movement is not
   your business) — with many metrics the response is heavy. Legacy experiments
   (`ExperimentTrendsQuery` / `ExperimentFunnelsQuery` metrics) aren't supported by this
   tool — fall back to the exposure SQL below.

Drop to `execute-sql` only for diagnosis: dating an onset, per-person fragmentation,
custom-exposure drill-downs. **Timezone footgun:** HogQL string timestamp literals parse
in the _project_ timezone, not UTC — a UTC `start_date` literal can shift the window by
hours and fake a dormant experiment. Use `now() - INTERVAL N DAY` for recency windows.

### Profile shape — config vs data

| Pattern                                                                             | What it usually means                                                       |
| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `sample_ratio_mismatch.p_value` < 0.01 at healthy volume                            | SRM — investigate first; this is the flagship finding                       |
| `$multiple` share > 0.5% of exposures (or > 0.1% with an uneven split + `exclude`)  | Identity fragmentation or mid-run rebucketing — contamination               |
| SRM clean but `multiple_variant_percentage` high                                    | The failure SRM alone misses — surviving arms balance, excluded users don't |
| Primary metric `data: null` or `validation_failures` in all arms, exposures healthy | Metric machinery broken — measuring nothing while burning decision time     |
| Running experiment, zero exposures in 48h after a healthy baseline                  | Dormant — flag call removed from code, or upstream broke                    |
| Running experiment, zero exposures ever, launched > 24h ago                         | Broken wiring — wrong SDK method, flag at 0%, custom exposure misconfigured |
| Flag `filters` edited after `start_date`                                            | Mid-run mutation — post-edit data may be contaminated                       |
| Running far past `recommended_running_time` with flat exposure accumulation         | Zombie — P3 recommendation to decide or end                                 |
| Stopped experiment, flag still active serving multiple variants weeks later         | Lingering contamination + flag debt — P3 hygiene                            |
| Ratio matches split, volume healthy, no recent flag edits                           | Baseline — leave it alone regardless of metric movement                     |

### Explore

Patterns to watch — starting points, not a checklist.

#### Sample ratio mismatch (SRM)

For each running experiment launched > 24h ago, read
`exposures.sample_ratio_mismatch.p_value` off `experiment-results-get` — PostHog runs the
chi-squared itself (`$multiple` excluded). p < 0.01 at healthy volume is the flag; cite
the p-value and per-variant `total_exposures` vs the `expected` counts in the finding.

Two caveats before trusting a clean p-value:

- It tests against the **current** configured split. If variants were redistributed
  mid-run, post-edit balance can l