beta-testing

What this skill does

# Beta Testing Strategy

End-to-end beta testing workflow for Apple platform apps — from TestFlight setup through feedback collection to launch readiness decision.

## When This Skill Activates

Use this skill when the user:
- Wants to run a beta test or plan a beta program
- Needs to set up TestFlight for internal or external testing
- Asks how to collect user feedback during beta
- Wants to know if their app is ready to launch
- Needs help designing a beta feedback survey
- Asks "how many beta testers do I need?"
- Mentions "TestFlight", "beta testers", "soft launch", or "launch readiness"
- Wants to plan a structured beta rollout
- Needs a go/no-go framework for shipping

## Process

### Phase 1: TestFlight Program Strategy

Before recruiting testers, understand Apple's two-tier beta system.

#### Internal Testing

| Attribute | Details |
|-----------|---------|
| Max testers | 100 |
| App Review required | No |
| Build availability | Immediate after upload |
| Tester requirement | Must be App Store Connect users |
| Best for | Team, close collaborators, developer friends |
| Expiration | 90 days from build upload |

**Use internal testing for:**
- Catching crashes and obvious bugs before anyone else sees the app
- Validating core flows work end-to-end
- Getting feedback from people who will be honest (friends, fellow devs)

#### External Testing

| Attribute | Details |
|-----------|---------|
| Max testers | 10,000 |
| App Review required | Yes (Beta App Review — usually 24-48 hours) |
| Build availability | After Beta App Review approval |
| Tester requirement | Anyone with an email address and iOS/macOS device |
| Best for | Real users, broader audience, validating product-market fit |
| Expiration | 90 days from build upload |

**Use external testing for:**
- Validating with real users who have no context about your app
- Stress-testing with diverse devices, OS versions, and usage patterns
- Gathering signal on whether the value proposition resonates

#### Group Management

Create tester cohorts for targeted feedback:

| Cohort | Size | Purpose | What to Ask |
|--------|------|---------|-------------|
| Power Users | 10-20 | Deep feature testing, edge cases | Does this handle your advanced workflows? |
| Casual Users | 20-50 | First-impression and onboarding quality | Was anything confusing in the first 5 minutes? |
| Accessibility Testers | 5-10 | VoiceOver, Dynamic Type, color contrast | Can you complete core tasks with accessibility features? |
| Domain Experts | 5-10 | Validate domain-specific correctness | Is the [domain] logic accurate and trustworthy? |

**TestFlight group tips:**
- Name groups clearly (e.g., "Wave 1 - Power Users", "Wave 2 - General")
- Use different "What to Test" descriptions per group
- Stagger group invitations — don't invite everyone at once

### Phase 2: Beta Tester Recruitment

#### How Many Testers Per Stage

| Stage | Testers | Duration | Goal |
|-------|---------|----------|------|
| Internal alpha | 10-20 | 1-2 weeks | Crash-free, core flows work |
| External wave 1 | 50-200 | 2 weeks | Validate UX, find confusion points |
| External wave 2 | 200-1,000 | 2 weeks | Stress-test, validate at scale |
| Open beta (optional) | 1,000+ | 1-2 weeks | Final validation, build buzz |

**Rule of thumb:** You need at least 50 external testers to get meaningful signal. Below 50, individual preferences dominate.

#### Where to Find Beta Testers

**High-quality sources (engaged, will give feedback):**
- **Indie dev communities** — Indie Dev Monday, iOS Dev Weekly Slack, Mastodon #iosdev
- **Reddit communities** — r/iOSBeta, r/apple, domain-specific subreddits (r/productivity, r/fitness, etc.)
- **Twitter/X** — Post "Looking for beta testers for [one-line pitch]" with a screenshot
- **ProductHunt Upcoming** — List your app, collect emails before launch
- **Friends and family** — Honest if you tell them honesty matters more than kindness

**Medium-quality sources (volume, less feedback):**
- **BetaList** — Submit for free listing
- **Beta testing communities** — BetaFamily, ErliBird
- **Discord servers** — App development and domain-specific servers

**Low-quality sources (avoid for signal):**
- Random giveaway sites (testers who just want free stuff)
- Paid beta testers (financially motivated, not genuine users)

#### Incentive Strategies

| Incentive | Best For | Notes |
|-----------|----------|-------|
| Early access | All testers | The default — being first is often enough |
| Lifetime free / pro unlock | Power users | Strong motivator, limited cost to you |
| Credit in app (About screen) | Engaged testers | Recognition matters to some users |
| Direct access to developer | Power users | They feel heard, you get deep feedback |
| Discount at launch | Wave 2+ testers | Good for larger cohorts |

**What NOT to offer:** Cash payment for testing. It attracts the wrong people and biases feedback.

### Phase 3: Feedback Collection Methodology

Collect feedback through multiple channels — different methods catch different signals.

#### Channel 1: In-App Feedback Form

Build a simple feedback mechanism directly in the app. Three fields are enough:

```
1. What's broken? (bugs, crashes, errors)
2. What's confusing? (UX that doesn't make sense)
3. What's missing? (features you expected but didn't find)
```

**Implementation tips:**
- Add a "Send Feedback" button in Settings (always visible during beta)
- Include device info, OS version, and app version automatically
- Attach a screenshot option (but don't require it)
- Send to a dedicated email or use a simple form service
- Timestamp and tag by tester cohort

#### Channel 2: TestFlight Built-In Feedback

TestFlight's native feedback is surprisingly useful:
- Users take a screenshot, annotate it, and add text
- Feedback arrives in App Store Connect with device/OS metadata
- Crash reports are automatic
- No extra infrastructure needed

**Tip:** In your TestFlight "What to Test" field, be specific:
```
This week, please test:
1. Creating a new [item] from scratch
2. Editing an existing [item]
3. Sharing [item] with someone

Report anything confusing or broken via TestFlight feedback (screenshot + description).
```

#### Channel 3: Short Surveys (Max 5 Questions)

Send a survey at the end of each beta wave. Use AskUserQuestion to help design survey questions tailored to the app.

**Template survey (adapt per app):**

1. **How would you rate the overall experience?** (1-5 stars)
2. **What was the most confusing part?** (free text)
3. **What feature would make you use this daily?** (free text)
4. **Would you pay for this app?** (Yes / No / Maybe — if Yes, how much?)
5. **How likely are you to recommend this to a friend?** (0-10 NPS scale)

**Survey rules:**
- Maximum 5 questions — completion rate drops 20% per additional question
- Always include one open-ended question (the best insights come from free text)
- Send via email, not in-app (don't interrupt usage)
- Send 5-7 days after they start testing (enough time to form opinions)

#### Channel 4: 1-on-1 User Interviews

The highest-signal feedback channel. Do 5-8 interviews per beta wave.

**Who to interview:**
- 2-3 testers who used the app heavily (understand power user needs)
- 2-3 testers who tried it once and stopped (understand drop-off reasons)
- 1-2 testers from the accessibility cohort

**Logistics:**
- 20-30 minutes via video call or phone
- Record with permission (for your notes, not public)
- Take written notes even if recording

### Phase 4: User Interview Guide

#### The 5 Key Questions

Ask these in order. Each builds on the previous:

1. **"What were you trying to do when you opened the app?"**
   - Reveals their mental model and expectations
   - Listen for: Does their goal match your intended use case?

2. **"Walk me through what happened."**
   - Have them narrate their experience step by step
   - Listen for: Where did they pause, backtrack, or get stuck?

3. **"What did you expect to happen at [specific moment