playtest-design

What this skill does

# Playtest Design

**Purpose:** Get useful signal from playtests. Most playtest sessions are wasted — observers confirm what they already believe, ask leading questions, and draw conclusions from noise. This skill provides structured methods to avoid those traps.

**Influences:** Frameworks here draw on cognitive UX research methodology, metrics-driven iterative design practice, and experience engineering theory (emergent behavior observation, planning under uncertainty).

---

## When to Activate

Use this skill when:
- Planning a playtest session (what to test, who to recruit, what to measure)
- Designing post-playtest surveys or interview questions
- Setting up analytics/metrics for ongoing data collection
- Interpreting playtest results and deciding what to change
- Resolving team disagreements about what the data shows

---

## Core Principle: Observe, Then Ask

Players are reliable reporters of their *experience* (what they felt) but unreliable reporters of *causes* (why they felt it). Design your process accordingly.

```
Most Reliable ←———————————————→ Least Reliable
  What they did    What they felt    Why they think
  (behavior)       (experience)      they felt it
                                     (attribution)
```

**Hierarchy of evidence:**
1. **Behavioral data** — what players actually did (metrics, video, observation)
2. **Experience reports** — what players say they felt ("I was frustrated," "that was exciting")
3. **Causal attribution** — what players think caused their experience ("the controls are bad")

Players attributing frustration to "bad controls" might actually be experiencing a perception failure (they couldn't see the indicator) or a pacing problem (too many new concepts at once). Use behavior to diagnose; use self-report to locate.

---

## Question Generation Framework

### The Three-Pillar Method

Generate questions along the perception → attention → memory pipeline:

**Perception Questions** (Did they see it?)
- Did the player notice [critical UI element / feedback / environmental cue]?
- How long before they noticed?
- Did they look at it before acting or after?
- Did they confuse it with something else?

**Attention Questions** (Did they focus on the right thing?)
- Where was the player looking during [critical moment]?
- Did they engage with [intended system] or get distracted by [ancillary system]?
- Did they understand what was important vs. optional?
- Was there a moment where they seemed overwhelmed?

**Memory Questions** (Will they retain it?)
- After a break, can they recall how to [key mechanic]?
- Did they apply a lesson learned earlier to a later challenge?
- Did they remember the goal after a distraction?
- Can they explain the core rules to someone else?

### Stage-Specific Questions

| Dev Stage | Focus | Key Questions |
|-----------|-------|---------------|
| **Prototype** | Core loop viability | Is the core action inherently interesting? Do they want to do it again? |
| **Alpha** | System comprehension | Do they understand the rules? Can they make intentional decisions? |
| **Beta** | Pacing and polish | Does the session arc feel right? Where do they get bored or frustrated? |
| **Pre-launch** | Edge cases and balance | What breaks? What's exploitable? What did we miss? |

---

## Observation Protocol

### What to Watch (Not Ask)

| Observable | What It Tells You |
|------------|-------------------|
| **First action** | What the UI communicates as "start here" |
| **Hesitation points** | Where clarity fails or cognitive load spikes |
| **Repeated failures** | Where difficulty exceeds skill (or UI is misleading) |
| **Where they look** | What's grabbing attention (intended or not) |
| **Body language** | Leaning in = engaged; leaning back = disengaged; fidgeting = frustrated |
| **Utterances** | Unprompted comments ("what?", "oh!", "come on") are gold |
| **Where they quit** | The most valuable data point you'll collect |
| **What they skip** | Content they ignore reveals priority mismatches |

### The Silent Observer Protocol

1. **Say nothing** unless they're about to break the test setup
2. **Don't explain** — if they're confused, that's data
3. **Don't reassure** — "you're doing great" biases the session
4. **Note timestamps** — when you feel the urge to help, write down the time and why
5. **Record everything** — your memory of the session will be biased toward your expectations

---

## Metrics to Track

### Core Metrics (Track Always)

| Metric | What It Measures | Warning Signal |
|--------|-----------------|----------------|
| **Session length** | Engagement | Bimodal distribution (some quit fast, some stay long) |
| **Quit points** | Pain points | Cluster of quits at same location/moment |
| **Completion rate** | Difficulty/clarity | < 70% on intended-critical-path content |
| **Time per section** | Pacing | Sections taking 2x+ longer than designed |
| **Death/failure rate** | Difficulty curve | Spike = wall; zero = too easy |

### Balance Metrics (When Tuning Systems)

| Metric | What It Measures | Warning Signal |
|--------|-----------------|----------------|
| **Pick rate by option** | Strategy diversity | One option > 50% pick rate |
| **Win rate by strategy** | Balance | Any strategy > 55% win rate at comparable skill |
| **Average game/match length** | Pacing | Games consistently shorter or longer than intended |
| **Resource accumulation rate** | Economy health | Exponential growth = inflation incoming |
| **Strategy churn** | Meta health | If dominant strategy shifts too fast, balance is noisy |

### UX Metrics (When Testing Comprehension)

| Metric | What It Measures | Warning Signal |
|--------|-----------------|----------------|
| **Time to first meaningful action** | Onboarding quality | > 60 seconds before the player *does* something |
| **Tutorial completion rate** | Tutorial design | < 90% = tutorial is the problem, not the player |
| **Hint/help usage** | Clarity | High usage = UI isn't communicating; zero usage = help system is invisible |
| **Error rate on intended actions** | Usability | Player tries to do the right thing but fails due to UI |

---

## Avoiding Confirmation Bias

The biggest threat to useful playtest data is your own expectations.

### Pre-Test Protocol

Before the session:
1. **Write down your predictions** — what do you expect to happen?
2. **Define "surprising" outcomes** — what would change your mind?
3. **Assign a skeptic** — one team member whose job is to challenge interpretations
4. **Pre-commit to sample size** — decide how many sessions before drawing conclusions (minimum 5 for qualitative, 30+ for quantitative)

### Post-Test Protocol

After the session:
1. **Review predictions vs. reality** — where were you wrong? Those are the insights.
2. **Separate observation from interpretation** — "Player hesitated for 8 seconds at the door" (observation) vs. "Player didn't understand the door mechanic" (interpretation)
3. **Look for disconfirming evidence** — actively search for data that contradicts your preferred narrative
4. **Quantify before concluding** — "it felt like everyone struggled" vs. "3 of 7 players failed this section"
5. **Delay solutions** — understand the problem fully before proposing fixes

### Common Bias Traps

| Trap | Mechanism | Counter |
|------|-----------|---------|
| **Anchoring** | First session dominates your impression | Review all sessions before concluding |
| **Availability** | Dramatic moments overshadow quiet ones | Use metrics, not memory |
| **Projection** | Attributing your own experience to players | Watch what they *do*, not what you'd do |
| **Sunk cost** | Defending features you spent time on | Ask "would we add this today?" not "should we cut this?" |
| **Survivorship** | Only hearing from players who stayed | Track quit points with equal priority |

---

## Survey Design

### Good Questions (Experience-Focused)

- "How would you describe the experience in one word?"
- "What moment stands out most?" (Then