AI-Powered Visual Regression Testing

What this skill does

# AI-Powered Visual Regression Testing

## Overview

Traditional visual regression testing produces overwhelming false positives from anti-aliasing, timestamps, and other noise. This skill implements **AI-powered visual regression** that understands the difference between intentional design changes and actual bugs.

**Key Innovation:** Uses Claude AI to analyze visual diffs with context awareness (git commits, design token changes, component history) to categorize changes intelligently.

## When to Use This Skill

Trigger this skill when the user:
- Mentions "visual regression testing" or "screenshot comparison"
- Wants to "detect UI changes" or "catch visual bugs"
- Says "pixel diff is too noisy" or "too many false positives"
- Asks to "set up visual testing" for their Storybook
- Wants to "review visual changes" in a PR
- Mentions Chromatic, Percy, or other visual testing tools

## Core Capabilities

### 1. Intelligent Diff Analysis

**Problem:** Traditional pixel diff flags thousands of irrelevant changes:
- Anti-aliasing differences
- Timestamp updates
- Random UUIDs in content
- Sub-pixel rendering variations

**Solution:** AI categorizes changes by semantic meaning:
- **Ignore:** Rendering noise, timestamps, random data
- **Expected:** Matches recent design system updates
- **Warning:** Significant but possibly intentional
- **Error:** Clear regressions (misalignment, broken layout)

### 2. Context-Aware Decision Making

The AI analyzer considers:
- **Git commits** (last 7 days) - Did we just update the theme?
- **Design tokens** - Does the new color match a token update?
- **Component history** - Was this component recently refactored?
- **PR description** - Did the developer mention this change?

### 3. Smart Auto-Approval

Define auto-approval rules:
- Approve all changes matching design token updates
- Approve timestamp/UUID changes
- Approve anti-aliasing differences
- Flag layout shifts for manual review

## Technical Implementation

### Architecture

```
1. Capture screenshots (baseline + current)
   ↓ Playwright/Storybook Test Runner
2. Generate pixel diff
   ↓ pixelmatch library
3. AI analysis with context
   ↓ Claude analyzes diff + git history + tokens
4. Categorize changes
   ↓ Ignore, Expected, Warning, Error
5. Generate actionable report
   ↓ With recommendations and auto-fix options
```

### Setup Command

Use `/setup-visual-testing` to configure:
- Installs @storybook/test-runner, Playwright
- Creates configuration files
- Captures initial baseline screenshots
- Sets up AI analysis pipeline
- Configures CI/CD integration (optional)

### Analysis Workflow

```typescript
// After code changes
npm run test:visual

// Output:
Running visual regression tests...
  ✓ 42 components: No changes
  ⚠️ 3 components: Potential regressions detected
  ❌ 2 components: Likely bugs found

AI Analysis Report:

Button Component:
  ⚠️ Color change detected: #2196F3 → #1976D2
  Context: Recent commit updated theme.ts (2 hours ago)
  Analysis: Matches new primary-600 token - appears intentional
  Recommendation: APPROVE (auto-approve with --accept-theme-changes)

Card Component:
  ❌ Layout shift: Content misaligned by 2.3px
  Context: No related changes in recent commits
  Analysis: Box-sizing or padding regression
  Recommendation: REJECT - needs investigation
  Git blame: Modified in commit def456 (unrelated refactor)

Modal Component:
  ⚠️ Shadow change: Elevation increased
  Context: Recent commit updated elevation system
  Analysis: Matches new shadow-lg definition
  Recommendation: APPROVE (design system update)
```

## Integration Points

### 1. Storybook Test Runner

```javascript
// .storybook/test-runner-config.ts
import { getStoryContext } from '@storybook/test-runner';
import { analyzeVisualDiff } from './visual-regression-ai';

export default {
  async postRender(page, context) {
    const storyContext = await getStoryContext(page, context);

    // Capture screenshot
    const screenshot = await page.screenshot();

    // Compare with baseline
    const diff = await compareWithBaseline(context.id, screenshot);

    if (diff.pixelsChanged > 0) {
      // AI analysis
      const analysis = await analyzeVisualDiff({
        diff,
        storyId: context.id,
        componentName: storyContext.component,
        recentCommits: await getRecentCommits(),
        designTokens: await loadDesignTokens()
      });

      // Categorize
      if (analysis.category === 'error') {
        throw new Error(analysis.message);
      } else if (analysis.category === 'warning') {
        console.warn(analysis.message);
      }
    }
  }
};
```

### 2. CI/CD Integration

```yaml
# .github/workflows/visual-regression.yml
name: Visual Regression Testing

on: [pull_request]

jobs:
  visual-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install dependencies
        run: npm ci
      - name: Build Storybook
        run: npm run build-storybook
      - name: Run visual regression tests
        run: npm run test:visual
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: visual-regression-report
          path: .storybook/visual-regression-report/
```

### 3. Local Development

```bash
# First time setup
/setup-visual-testing

# After making changes
npm run test:visual

# Auto-approve theme changes
npm run test:visual -- --accept-theme-changes

# Interactive mode (review each change)
npm run test:visual -- --interactive

# Update baselines
npm run test:visual -- --update-baselines
```

## AI Analysis Logic

### Change Classification

```python
# skills/visual-regression-testing/scripts/analyze_diff.py

def categorize_change(change, context):
    """Categorize a visual change using AI analysis"""

    # 1. Check if change is just rendering noise
    if is_rendering_noise(change):
        return Category.IGNORE, "Anti-aliasing or sub-pixel rendering"

    # 2. Check if change matches design token update
    if matches_design_token_update(change, context.design_tokens):
        token = find_matching_token(change, context.design_tokens)
        return Category.EXPECTED, f"Matches {token} update in recent commit"

    # 3. Check if change was mentioned in PR/commit
    if mentioned_in_commits(change, context.recent_commits):
        return Category.EXPECTED, "Change mentioned in commit message"

    # 4. Analyze semantic significance
    if is_layout_shift(change):
        # Layout shifts are almost always bugs
        return Category.ERROR, "Layout misalignment detected"

    if is_color_change(change):
        # Color change without token update = warning
        return Category.WARNING, "Color changed but not in design tokens"

    if is_typography_change(change):
        # Typography change = warning
        return Category.WARNING, "Typography change detected"

    # 5. Default to warning for significant changes
    if change.pixels_changed > threshold:
        return Category.WARNING, "Significant visual change, please review"

    return Category.IGNORE, "Minor change within acceptable threshold"
```

### Context Analysis

```python
def analyze_with_context(diff_image, baseline_image, context):
    """Analyze diff with full context awareness"""

    # Load context
    recent_commits = get_git_commits(days=7)
    design_tokens = load_design_tokens()
    component_history = load_component_history(context.component_name)

    # Compute pixel diff
    pixel_changes = compute_pixel_diff(baseline_image, diff_image)

    # Cluster changes by type
    color_changes = extract_color_changes(pixel_changes)
    position_changes = extract_position_changes(pixel_changes)
    size_changes = extract_size_changes(pixel_changes)
    text_changes = extract_text_changes(pixel_changes)

    # Analyze each cluster
    categorizations = []

    for change in color_changes:
        category, reason = categorize_co