analyze-experiments

What this skill does

# Experiment Analyst

Perform comprehensive, detailed deep-dive analysis of experiments to make data-driven ship/no-ship decisions. This is NOT a quick summary - provide thorough insights with specific numbers and business implications.

## When to Use

- Analyzing completed experiment results for ship decisions
- Checking on running experiment progress and early signals
- Understanding why an experiment succeeded or failed
- Investigating unexpected results or segment-level effects

---

## Analysis Philosophy

**Be comprehensive, not brief:**
- Include specific numbers, percentages, and data points
- Explain statistical meaning AND business implications in plain language
- Cover all metrics (primary, secondary, guardrails) with actual values
- This is a single comprehensive analysis - do not rush or provide superficial summaries

---

## Instructions

### Step 0: Identify Experiment

**If user provides a specific experiment:**
- Accept experiment **URL or experiment ID**
- If URL: use `Amplitude:get_from_url` to extract details
- If ID: proceed to Step 1

**If user asks about experiments generally:**
- Use `Amplitude:search` with `entityTypes: ["EXPERIMENT"]` and relevant query terms
- Present top 3-5 matches with names, IDs, and states
- Ask user which experiment to analyze

**If no experiment specified:**
- Ask explicitly for experiment URL, ID, or search terms and stop

---

### Step 1: Retrieve and Validate Setup

Use `Amplitude:get_experiments` with experiment ID to capture:

- Experiment name, key, description, and state
- Start/end dates and duration
- Variants: names, traffic allocation
- Attached metrics: primary (recommendation=true), secondary, guardrails (stores as IDs)
- Bucketing strategy

**Get metric names:**
- Extract metric IDs from the experiment response (e.g., "c4pn8fkv")
- **CRITICAL: Amplitude MCP cannot retrieve metric names by ID directly**
- Workaround options:
  1. Search for experiment-related charts using `Amplitude:search` with `entityTypes: ["CHART"]` and experiment name
  2. Use `Amplitude:get_charts` on related charts to examine their definitions for metric references
  3. Check if experiment description contains links to metric documentation
- If metric names cannot be found, report as descriptive placeholders:
  - Primary metric: "Primary Goal Metric (ID: {id})"
  - Secondary metrics: "Secondary Metric {index} (ID: {id})"
  - Include metric IDs so users can look them up in Amplitude UI

**Validation:**
- Is experiment running or completed? (not draft)
- Has it run for 1+ weeks?
- Are variants and metrics clearly defined?

If incomplete, explain what's missing and stop.

---

### Step 2: Check Data Quality (with explicit thresholds)

Use `Amplitude:query_experiment` (primary metric only) to assess:

**Traffic Balance (SRM Check):**
- Report actual traffic split per variant (e.g., 48.2% control, 51.8% treatment)
- **Use `srmDetected` field from API:** Flag if `srmDetected: true`
- SRM (Sample Ratio Mismatch) indicates the observed traffic split deviates significantly from the expected allocation
- If SRM detected, report the expected vs. actual allocation with specific percentages
- Severe SRM can indicate instrumentation issues or bucketing problems that may invalidate results

**Sample Size Analysis:**

**A. Current Sample Assessment:**
- Report total users per variant with specific numbers
- **Flag if <100 users per variant** (insufficient for any conclusion)
- **Flag if 100-1000 users** (directional signals only, not confident decision)
- Need 1000+ per variant for confident decisions

**B. Statistical Power Analysis:**
- **Target effect size:** What minimum lift would be meaningful for the business? (typically 2-5% for conversion metrics)
- **Achieved power:** Given current sample size and observed variance, what's the probability of detecting the target effect if it exists?
- **Power interpretation:**
  - <50%: Severely underpowered - likely to miss real effects
  - 50-70%: Underpowered - high risk of false negatives
  - 70-80%: Marginally adequate - consider extending if p-value is borderline
  - 80%+: Well-powered - sufficient to detect target effect size
- **If underpowered:** Calculate additional sample size needed to reach 80% power
- **Recommendation:** If power <70% and results are inconclusive, extend duration rather than making premature decision

**C. Precision Analysis (Confidence Interval Width):**
- **CI width for primary metric:** Report the width of the 95% confidence interval as percentage of baseline
- **Precision assessment:**
  - CI width >10% of baseline: Low precision - effect size uncertainty too high for confident decisions
  - CI width 5-10% of baseline: Moderate precision - acceptable for directional decisions
  - CI width <5% of baseline: High precision - narrow enough for confident decisions
- **Actionability threshold:** Is the CI narrow enough to distinguish between practically significant and negligible effects?
  - If lower CI bound suggests meaningful lift but upper bound is marginal, precision may be insufficient
  - Example: If target is +5% lift and CI is [-2%, +12%], too wide to confidently conclude effect exceeds target
- **Recommendation:** If CI too wide, extend duration or increase traffic allocation to improve precision

**Comprehensive Data Quality Flags:**

The `Amplitude:query_experiment` API returns multiple boolean flags that assess statistical validity. Check and document each:

1. **statsAssumptionsMetForWholeExperiment:**
   - Indicates whether core statistical assumptions are satisfied (normality, independence)
   - **If false:** Results may not be reliable; consider non-parametric approaches or longer runtime
   - Impact: High - affects all statistical conclusions

2. **hasSuspiciousUplift:**
   - Flags unexpectedly large effect sizes that may indicate data quality issues
   - **If true:** Verify instrumentation, check for bot traffic, or segment anomalies
   - Impact: High - may indicate measurement error rather than real effect

3. **isVariancePositive:**
   - Confirms metric variance is positive (mathematically required for statistical tests)
   - **If false:** Critical data quality issue - metric may be constant or incorrectly computed
   - Impact: Critical - statistical tests invalid if false

4. **isConfidenceIntervalNotFlipped:**
   - Ensures lower CI bound < upper CI bound (mathematical consistency check)
   - **If false:** Indicates calculation error or data corruption
   - Impact: Critical - results cannot be trusted

5. **isStandardErrorLargeEnough:**
   - Checks if standard error is sufficient for reliable inference
   - **If false:** High variance or very small sample may produce unreliable confidence intervals
   - Impact: Medium - affects precision of estimates

6. **isPointEstimateInsideConfidenceInterval:**
   - Validates that point estimate falls within its confidence interval (consistency check)
   - **If false:** Calculation error or numerical instability
   - Impact: High - indicates statistical computation issues

7. **isMeanValid:**
   - Confirms mean value is a valid number (not NaN, not infinite)
   - **If false:** Data quality issue - check for null values or computation errors
   - Impact: Critical - cannot analyze if mean is invalid

**For each flag that fails (returns false or true for suspicious uplift), document:**
- Which flag failed
- What it means in plain language
- Specific impact on result reliability
- Recommended action (extend duration, investigate instrumentation, etc.)

**If all flags pass:** Note this explicitly as strong data quality signal

**Temporal Stability:**
- Check if primary metric is stable day-over-day
- Note ramp period (first 24-48hrs) or day-of-week effects

**Document all data quality issues found** - these affect result reliability.

---

### Step 3: Analyze Primary Metric

Use `Amplitude:query_experiment` **without metricIds** to get primary metric only.

**Use metric name from Step 1** - Report u