review-change

What this skill does

# /marketplace:review-change — Pre-Ship Review of a Relevance or Personalisation Change

> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).

Stress-test a proposed change to search or personalisation **before** it ships. Pulls the golden set, runs offline eval, walks the rule checklist, checks observability for blind spots, and recommends a ship decision.

## Usage

```
/marketplace:review-change <free-form description of the change, or a PR link>
```

Examples:

```
/marketplace:review-change adding synonym map "doggy -> dog" and expanding "weekend" to "(friday OR saturday OR sunday)"
/marketplace:review-change swapping homefeed recipe from user-personalization-v1 to user-personalization-v2
/marketplace:review-change adding serving-time filter "cap-provider-4pct" to homefeed
/marketplace:review-change https://github.com/company/repo/pull/1234
```

## Workflow

### 1. Load Company Context

Read the `<company>-marketplace-context` skill:
- `indexes.md` — if the change touches `~~search engine`
- `recipes.md` — if the change touches `~~personalisation engine`
- `golden-set.md` — the offline eval corpus
- `observability.md` — the monitors that should catch a regression
- `gotchas.md` — any prior incidents matching the change type

### 2. Classify the Change

| Type | Signal |
|------|--------|
| **`~~search engine` relevance** | synonym, analyzer, mapping, BM25, LTR model, query rewrite, filter clause movement, boost weight |
| **`~~personalisation engine` change** | recipe swap, dataset schema, filter DSL, re-ranker, cold-start solution, feature set |
| **Observability / infrastructure** | monitor threshold, alert routing, dashboard only — route to `/marketplace:build-observability` instead |

If ambiguous, ask. If the change touches both types, split the review into two passes.

### 3. Branch A — Search Change

#### 3a. Summarise the change

Restate the change in structured form:
- **File(s) affected**: e.g., analyzer JSON, query builder, mapping template
- **Scope**: which index(es), which field(s), which query type
- **Reversibility**: trivial to roll back, or requires re-index?

#### 3b. Run offline eval against the golden set

Load `golden-set.md`. If empty or thin, advise running `/marketplace:build-golden-set search` first.

**If `~~search engine` is connected:**
1. Execute each golden query against the current production settings
2. Execute each against a staging clone or point-in-time snapshot with the proposed change applied
3. Compute per-query metrics: nDCG@10, MRR, zero-result rate, position delta for must-appear items
4. Report per-query diff, sorted by worst regression

**If not connected:**
Advise running the diff manually, and describe the minimal shape of the comparison the user should run.

#### 3c. Walk the rule checklist

Apply these checks (from `marketplace-search-recsys-planning`):

| Rule | Applies when | Check |
|------|-------------|-------|
| `query-curate-synonyms-by-domain` | synonym change | Are the synonyms validated per intent class? |
| `index-match-index-and-query-time-analyzers` | analyzer change | Do both sides use the same analyzer? |
| `rank-tune-bm25-parameters-last` | BM25 parameter change | Is the golden-set effect measured before rolling out? |
| `rank-normalise-scores-across-retrieval-primitives` | score change | Are scores from different primitives normalised? |
| `retrieve-use-filter-clauses-for-exact-matches` | filter→must movement | Does the change lose a hard constraint? |
| `retrieve-use-bool-structure-deliberately` | bool rewrite | Is the `should` / `must` / `filter` split explicit? |
| `query-use-fuzzy-matching-for-typos` | fuzzy change | Is the fuzziness budget bounded? |
| `query-normalise-before-anything-else` | any query-time change | Is normalisation still earliest in the pipeline? |
| `index-use-index-templates-for-consistency` | mapping change | Does the change propagate via template? |
| `measure-track-ndcg-mrr-zero-result-rate` | any | Will the deployed system track the metrics before / after? |

#### 3d. Check observability

From `observability.md`:
- Which monitors would catch a regression within 6 hours?
- Are they currently green? Any pre-existing alerts that would confuse the change's effect?
- Is SLO error budget sufficient to absorb a partial regression?

#### 3e. Produce review verdict

Structured output:

```markdown
## Review: {{change description}}

### Summary
{{1-paragraph restatement}}

### Golden-set diff
- **Aggregate**: nDCG@10 {{delta}}, MRR {{delta}}, zero-result {{delta}}
- **Worst regressions** ({{n}} queries):
  - {{query}}: nDCG {{old → new}} — {{suspected cause}}
  - ...
- **Improvements** ({{n}} queries):
  - ...

### Rule checklist
- [✓] {{rule}} — {{notes}}
- [✗] {{rule}} — {{issue + remediation}}

### Observability
- **Monitors that would catch regression**: {{list}}
- **SLO error budget**: {{state}}
- **Blind spots**: {{list}}

### Verdict
{{ship | ship-with-A/B | partial-rollback | no-ship}}

### Recommended next step
{{specific action}}

### Gotcha log entry (draft)
{{1-line entry for gotchas.md if this ships and something goes wrong}}
```

### 4. Branch B — Personalisation Change

#### 4a. Summarise the change

Restate in structured form:
- **Target**: dataset, solution, filter, re-ranker, feature, cold-start fallback
- **Scope**: which surfaces the change affects
- **Reversibility**: instant rollback (campaign / solution switch), re-train required, or requires dataset rebuild?

#### 4b. Sanity-check against the library

Apply these checks (from `marketplace-personalisation`):

| Rule | Applies when | Check |
|------|-------------|-------|
| `schema-design-conservatively` | schema change | Does the new schema deprecate a field safely? |
| `schema-meet-minimum-dataset-sizes` | recipe / dataset change | Are the minimum dataset size requirements still met? |
| `schema-include-context-everywhere` | schema change | Is the context propagated to training and serving? |
| `schema-prefer-categorical-fields` | feature change | Are features categorical where appropriate? |
| `recipe-default-to-user-personalization-v2` | recipe swap | Is the new recipe the right choice for this surface's intent? |
| `recipe-sims-for-item-page-only` | surface change | Is the recipe matched to the surface's job? |
| `recipe-defer-hpo-until-baseline-measured` | HPO change | Is a baseline measured first? |
| `cold-tag-cold-start-recs` | cold-start fallback change | Are cold-start impressions tagged for downstream analysis? |
| `cold-reserve-exploration-slots` | ranker change | Does exploration get reserved slots? |
| `loop-detect-death-spirals` | ranker change | Is the monitor in place? |
| `loop-reserve-random-exploration` | ranker change | Is there a serving-time exploration reserve? |
| `loop-optimize-completed-outcome` | label change | Is the ranking label the completed outcome (e.g., booking), not an intermediate (e.g., click)? |
| `match-balance-supply-demand` | ranker change | Does the ranker respect feasibility? |
| `match-cap-provider-exposure` | ranker change | Is there a single-supplier cap? |
| `infer-use-filters-api` | filter change | Is the filter using the managed API and not a hand-rolled rewrite? |
| `infer-cache-responses-short-ttl` | serving change | Is the TTL conservative enough to avoid staleness? |
| `track-use-stable-opaque-item-ids` | ID change | Are item IDs stable and opaque? |
| `track-stream-events-via-putevents` | event change | Is streaming used for live-user signals? |
| `track-capture-negative-signals` | event change | Are negative signals captured? |
| `obs-slice-metrics-by-segment` | observability | Are metrics decomposed by segment? |

**If the change introduces or modifies a feature** (text, vision, wizard-sourced, structured, or derived), also apply these rules from `marketplace-recsys-feature-engineering`:

| Rule | Applies when | Check |
|------|-------------|-------|
| `firstp-st