growth-experimenter

What this skill does

# Growth Experimenter

Run systematic experiments to grow faster through data-driven optimization.

## Core Philosophy

**Growth = Experimentation Velocity × Win Rate × Impact per Win**

- Run more experiments
- Increase your hit rate through better hypotheses
- Focus on high-impact areas

## Growth Model (AARRR / Pirate Metrics)

```
Acquisition → Activation → Retention → Revenue → Referral
    ↓             ↓            ↓          ↓         ↓
  Traffic      Sign Up      Day 30    Upgrade   Invites
  100%          40%          50%        20%       10%

Example: 10,000 visitors/month
→ 4,000 signups (40%)
→ 2,000 active at D30 (50%)
→ 400 paying (20%)
→ 40 referrals (10%)

Improve ANY metric by 10% = 10% more customers
```

**Where to focus first**: The leakiest bucket

- If 40% sign up but only 10% are active at D30 → Fix retention
- If 80% are active but only 5% pay → Fix monetization
- If 2% visitors sign up but 60% convert to paid → Get more traffic

## Experiment Framework

### 1. Identify the Problem

**Good problem statements**:

- "Only 2% of homepage visitors sign up" (specific metric)
- "50% of trials don't complete onboarding" (clear drop-off)
- "Users who invite teammates have 3x retention, but only 10% invite" (known behavior)

**Bad problem statements**:

- "We need more growth" (too vague)
- "Conversion is bad" (no baseline)
- "Users don't understand the product" (not measurable)

### 2. Form a Hypothesis

**Hypothesis template**:

```
We believe that [change]
will result in [outcome]
because [reason/evidence]
```

**Examples**:

```markdown
✅ Good:
We believe that adding social proof (testimonials) to the pricing page
will increase trial signups by 10%
because visitors currently have low trust and need validation.

✅ Good:
We believe that sending a Slack notification when user completes setup
will increase D7 activation by 20%
because users forget to come back after initial signup.

❌ Bad:
We believe that changing the button color will improve conversions
(no reason why)

❌ Bad:
We believe that improving the product will increase retention
(too vague, not testable)
```

### 3. Design the Experiment

**Experiment specification**:

```yaml
Experiment: Add social proof to pricing page

Hypothesis: Social proof on pricing will increase signups by 10%

Variants:
  Control: Current pricing page (no testimonials)
  Treatment: Pricing page + 3 customer testimonials

Primary Metric: Trial signup rate
Secondary Metrics:
  - Time on page
  - Scroll depth
  - CTA click rate

Sample Size: 1,000 visitors per variant
Duration: 2 weeks (or until statistical significance)
Success Criteria: >5% improvement with 95% confidence

Measurement:
  - Google Analytics
  - Mixpanel conversion tracking
  - Segment for event data
```

### 4. Run the Experiment

**A/B testing checklist**:

- [ ] Random assignment (50/50 split)
- [ ] Same time period (no day-of-week effects)
- [ ] Sufficient sample size
- [ ] No peeking (wait for significance)
- [ ] One change at a time

**Statistical significance calculator**:

```javascript
// Minimum sample size for 95% confidence
function calculateSampleSize(baseline, mde, power = 0.8, alpha = 0.05) {
  // baseline = current conversion rate (e.g., 0.02)
  // mde = minimum detectable effect (e.g., 0.10 for 10% lift)
  // Returns: visitors needed per variant

  const z_alpha = 1.96 // 95% confidence
  const z_power = 0.84 // 80% power

  const p1 = baseline
  const p2 = baseline * (1 + mde)
  const p_avg = (p1 + p2) / 2

  const n = (2 * p_avg * (1 - p_avg) * (z_alpha + z_power) ** 2) / (p2 - p1) ** 2

  return Math.ceil(n)
}

// Example: 2% baseline, detect 10% improvement
calculateSampleSize(0.02, 0.1) // ~35,000 visitors per variant
```

### 5. Analyze Results

**Interpreting results**:

```yaml
Control: 1,000 visitors → 20 conversions (2.0%)
Treatment: 1,000 visitors → 25 conversions (2.5%)

Lift: +25% relative (+0.5% absolute)
P-value: 0.04 (statistically significant if <0.05)
Confidence Interval: [-0.2%, +1.2%]

Decision: WIN - Ship it!
```

**When results are inconclusive**:

- **No movement**: Hypothesis was wrong or change too small
- **Not significant**: Need more data or larger effect
- **Negative impact**: Roll back immediately
- **Contradictory secondary metrics**: Investigate trade-offs

### 6. Scale What Works

```javascript
// After successful experiment, roll out to 100%
if (experimentResult.lift > 0.05 && experimentResult.pValue < 0.05) {
  rolloutFeature({
    feature: 'social_proof_on_pricing',
    rollout: '100%',
    monitor: ['signup_rate', 'trial_starts']
  })

  // Log the learning
  logExperimentLearning({
    learning: 'Social proof increased signups by 25%',
    application: 'Add social proof to all high-intent pages'
  })
}
```

## Growth Experiments by Stage

### Acquisition Experiments

**Goal**: Get more traffic or improve traffic quality

**High-impact experiments**:

1. **Landing page optimization**:

```yaml
Control: Generic homepage
Test: Tailored landing pages by traffic source
  - /for-startups (Product Hunt traffic)
  - /for-agencies (Google Ads)
  - /for-developers (GitHub referrals)

Expected lift: 20-50% on signup rate
```

2. **Headline testing**:

```yaml
Current: 'Project Management Software'
Test A: 'Ship Projects 2x Faster'
Test B: 'The Project Management Tool Teams Love'
Test C: "Finally, Project Management That Doesn't Suck"

Test: Value prop clarity, specificity, emotion
Expected lift: 10-30% on engagement
```

3. **Social proof**:

```yaml
Current: No social proof
Test: Add testimonials, logos, user count
  - "Join 10,000+ teams..."
  - Customer logos (recognizable brands)
  - Video testimonial from power user

Expected lift: 15-25% on trust/signups
```

### Activation Experiments

**Goal**: Get users to "aha moment" faster

**High-impact experiments**:

1. **Onboarding simplification**:

```yaml
Current: 7-step onboarding flow
Test: 3-step flow, delay advanced setup
  Step 1: Name + email
  Step 2: Create first project
  Step 3: Invite team (optional, skippable)

Expected lift: 30-50% completion rate
```

2. **Time-to-value reduction**:

```yaml
Current: Users must create project from scratch
Test: Pre-populated template
  - Sample project with tasks
  - Example data to explore
  - Guided tutorial

Expected lift: 25-40% in D1 activation
```

3. **Progress indicators**:

```yaml
Current: No feedback during setup
Test: Progress bar + completion checklist
  [✓] Account created
  [✓] First project
  [ ] Invite teammates (2 left)
  [ ] Complete first task

Expected lift: 15-25% completion rate
```

### Retention Experiments

**Goal**: Keep users coming back

**High-impact experiments**:

1. **Email re-engagement**:

```yaml
Current: No emails after signup
Test: 3-email onboarding sequence
  Day 1: "Here's how to get started"
  Day 3: "Tips from power users"
  Day 7: "You're only 1 step away from [value]"

Expected lift: 20-35% in D30 retention
```

2. **Habit building**:

```yaml
Current: No reminders
Test: Daily digest email
  "Your daily update: 3 tasks due today"
  - Creates daily habit
  - Drives return visits

Expected lift: 25-40% in daily active users
```

3. **Feature discovery**:

```yaml
Current: All features visible, overwhelming
Test: Progressive disclosure
  - Week 1: Core features only
  - Week 2: Unlock integrations
  - Week 3: Unlock advanced features
  - Tooltip hints for new features

Expected lift: 15-25% feature adoption
```

### Revenue Experiments

**Goal**: Convert free users to paying customers

**High-impact experiments**:

1. **Paywall optimization**:

```yaml
Current: Hard limit at 5 projects
Test: Soft limit + banner
  "You've created 5 projects! Upgrade to Pro for unlimited"
  - Allow them to continue
  - Show banner on every page
  - Show upgrade modal on 6th project

Expected lift: 20-30% in upgrade rate
```

2. **Trial length**:

```yaml
Current: 14-day trial
Test A: 7-day trial (more urgency)
Test B: 30-day trial (more time to get hooked)
Test C: U