usability-tester

What this skill does

# Usability Tester

Validate that users can successfully complete core tasks through systematic observation.

## Core Principle

**Watch users struggle.** The best way to find UX issues is to observe real users attempting real tasks. Their struggles reveal truth that surveys and analytics cannot.

## Test Planning

### 1. Define Test Objectives

```yaml
Good Objectives:
  - "Can users complete onboarding in <5 minutes?"
  - "Can users find and use the export feature?"
  - "Do users understand the pricing page?"

Bad Objectives:
  - "Test the UI" (too vague)
  - "See if users like it" (subjective, not behavioral)
```

### 2. Research Questions

```yaml
Examples:
  - Where do users get stuck during sign-up?
  - Can users find the settings page?
  - Do users understand what each tier includes?
  - What errors do users encounter?
```

### 3. Identify Core Tasks

Choose 3-5 tasks that represent key user journeys:

```yaml
Example Tasks (Project Management Tool): 1. Sign up and create account
  2. Create your first project
  3. Invite a team member
  4. Assign a task to someone
  5. Export project data
```

### 4. Recruit Participants

```yaml
Sample Size:
  - 5-8 users per persona
  - After 5 users, diminishing returns (Nielsen's research)
  - Test in waves: 5 users → fix issues → test 5 more

Recruitment Criteria:
  - Match target persona
  - Haven't used product before (for onboarding tests)
  - Or: Active users (for feature tests)

Incentives:
  - $50-100 per hour (B2C)
  - $100-200 per hour (B2B professionals)
  - Gift cards work well
```

## Task Scenarios

### Best Practices

✅ **Good task scenario**:

```
"Your team is launching a new project next week. Create a project
called 'Q2 Launch' and invite [email protected] to collaborate."
```

**Why it works**:

- Realistic context
- Clear goal
- Natural language
- Doesn't give step-by-step instructions

❌ **Bad task scenario**:

```
"Click the 'New Project' button, then enter 'Q2 Launch', then
click Settings, then click Invite, then enter email."
```

**Why it fails**:

- Step-by-step instructions
- No context
- Doesn't test discoverability
- User just follows orders

### Task Scenario Template

```yaml
Scenario: [Context/Motivation]
Goal: [What they need to accomplish]
Success Criteria: [How to know they completed it]

Example:
  Scenario: You're preparing for a client meeting tomorrow and need to review past conversations.
  Goal: Find all conversations with "Acme Corp" from the last 30 days
  Success Criteria: User successfully uses search/filter to find conversations
```

## Conducting Tests

### Think-Aloud Protocol

**Key instruction to participant**:

```
"Please think aloud as you work. Tell me what you're looking for,
what you're thinking, what you're trying to do. There are no
wrong answers - we're testing the product, not you."
```

**What to listen for**:

- "I'm looking for..." (what they expect)
- "I thought this would..." (mental models)
- "This is confusing because..." (friction points)
- "I'm not sure if..." (uncertainty)

### Facilitation Rules

✅ **Do**:

- Observe silently
- Take notes
- Let them struggle (reveals issues)
- Ask follow-up questions AFTER task
- Stay neutral

❌ **Don't**:

- Help or explain
- Lead them ("maybe try clicking...")
- Defend design choices
- Interrupt during task
- Show frustration

### Questions to Ask After Each Task

```yaml
Completion Questions:
  - 'On a scale of 1-5, how easy was that task?'
  - 'What were you expecting to see?'
  - 'What was confusing about that?'
  - 'If you could change one thing, what would it be?'

Discovery Questions:
  - 'Where did you expect to find that?'
  - 'What do you think this [feature] does?'
  - 'Why did you click there?'
```

## Metrics to Track

### Task Success Rate

```yaml
Measurement:
  - Completed: User achieved goal without help
  - Partial: User achieved goal with hints
  - Failed: User could not complete task

Calculation: Task Success Rate = (Completed Tasks / Total Attempts) × 100

Target: ≥80% for core tasks
```

### Time on Task

```yaml
Measurement:
  - Start timer when task begins
  - Stop when user completes or gives up

Analysis:
  - Compare to baseline/previous tests
  - Identify outliers (very fast or very slow)

Target: Varies by task complexity
  - Simple task (e.g., log in): <30 seconds
  - Medium task (e.g., create project): 1-2 minutes
  - Complex task (e.g., configure integration): 3-5 minutes
```

### Error Rate

```yaml
Errors:
  - Wrong path taken
  - Incorrect button clicked
  - Had to backtrack
  - Gave up and tried different approach

Calculation: Errors per Task = Total Errors / Number of Users

Target: <2 errors per task
```

### Satisfaction Rating

```yaml
Post-Task Question:
  "How satisfied are you with completing this task?" (1-5 scale)

  1 = Very Dissatisfied
  2 = Dissatisfied
  3 = Neutral
  4 = Satisfied
  5 = Very Satisfied

Target: ≥4.0 average
```

## Issue Severity Rating

### Severity Formula

```
Severity = Impact × Frequency
```

### Impact Scale (1-3)

```yaml
1 - Low Impact:
  - Minor inconvenience
  - User can easily recover
  - Cosmetic issue

2 - Medium Impact:
  - Causes delay or confusion
  - User eventually figures it out
  - Moderate frustration

3 - High Impact:
  - Blocks task completion
  - User cannot proceed without help
  - Critical to core functionality
```

### Frequency Scale (1-3)

```yaml
1 - Rare:
  - Only 1-2 users encountered
  - Edge case
  - Specific conditions

2 - Occasional:
  - 3-5 users encountered
  - Somewhat common
  - Specific user types

3 - Frequent:
  - Most/all users encountered
  - Consistent issue
  - All user types
```

### Combined Severity

```yaml
Critical (8-9):
  - Impact: 3, Frequency: 3
  - Blocks most users
  → Fix immediately before release

High (6-7):
  - Impact: 3, Frequency: 2 OR Impact: 2, Frequency: 3
  - Significant delay or frequent minor issue
  → Fix before release

Medium (4-5):
  - Impact: 2, Frequency: 2 OR Impact: 3, Frequency: 1
  - Minor frustration or rare blocker
  → Fix in next release

Low (1-3):
  - Impact: 1, Frequency: 1-3
  - Cosmetic or rare minor issue
  → Backlog
```

## System Usability Scale (SUS)

**10-question survey** (post-test, 1-5 Likert scale):

```yaml
Questions (Odd = Positive, Even = Negative): 1. I think I would like to use this product frequently
  2. I found the product unnecessarily complex
  3. I thought the product was easy to use
  4. I think I would need support to use this product
  5. I found the various functions well integrated
  6. I thought there was too much inconsistency
  7. I imagine most people would learn this quickly
  8. I found the product cumbersome to use
  9. I felt very confident using the product
  10. I needed to learn a lot before getting going

Scoring:
  - Odd questions: Score - 1
  - Even questions: 5 - Score
  - Sum all scores
  - Multiply by 2.5
  - Result: 0-100 score

Interpretation:
  ≥80: Excellent
  68-79: Good (industry average)
  51-67: OK
  <51: Needs significant improvement
```

## Test Report Template

```yaml
usability_test_summary:
  date: '2024-01-20'
  participants: 8
  participant_profile: 'New users, age 25-45, tech-savvy'

  tasks:
    - task: 'Create a new project'
      success_rate: '87.5% (7/8)'
      avg_time: '1m 24s'
      errors: 1.2 per user
      satisfaction: 4.3/5

    - task: 'Invite team member'
      success_rate: '62.5% (5/8)'
      avg_time: '2m 45s'
      errors: 2.8 per user
      satisfaction: 3.1/5

  issues:
    - issue: "Users can't find 'Invite' button"
      severity: high
      impact: 3
      frequency: 3
      affected_users: 7/8
      recommendation: "Move 'Invite' button to top of project page, make it more prominent"

    - issue: 'Confusion about project vs workspace'
      severity: medium
      impact: 2
      frequency: 3
      affected_users: 6/8
      recommendation: 'Add tooltip explaining difference, update onboarding'

    - issue: 'Export button text unclear'
      severity: low
      i