regression-baseline

What this skill does

# regression-baseline

Create and maintain regression test baselines for comparison and drift detection.

## Triggers


Alternate expressions and non-obvious activations (primary phrases are matched automatically from the skill description):

- "snapshot tests" → baseline creation shorthand
- "capture current behavior" → baseline establishment

## Purpose

This skill manages regression baselines by:
- Capturing known-good system states as baselines
- Storing snapshots of expected outputs
- Maintaining versioned baseline history
- Comparing current state to baseline
- Detecting drift from established baselines
- Managing baseline updates and approvals

## Behavior

When triggered, this skill:

1. **Identifies baseline scope**:
   - Determine what to baseline (tests, outputs, performance)
   - Select baseline type (functional, visual, performance, API)
   - Identify affected components
   - Check for existing baselines

2. **Captures baseline data**:
   - Run tests and capture outputs
   - Take snapshots (visual, data, API responses)
   - Measure performance metrics
   - Record system configuration
   - Generate checksums/hashes

3. **Stores baseline**:
   - Version baseline with semantic versioning
   - Tag with git commit/release
   - Store in `.aiwg/testing/baselines/`
   - Update baseline manifest
   - Link to requirements/features

4. **Validates baseline quality**:
   - Ensure baseline is stable (run multiple times)
   - Verify baseline is reproducible
   - Check for known issues in baseline
   - Require approval for critical baselines

5. **Documents baseline**:
   - Record what is baselined
   - Note when and why baseline was created
   - Document known deviations
   - Link to issues/requirements

6. **Manages baseline lifecycle**:
   - Archive old baselines
   - Update baselines on intentional changes
   - Track baseline drift over time
   - Alert on excessive drift

## Baseline Types

### Functional Baseline

```yaml
functional_baseline:
  description: Expected behavior outputs
  scope: Unit and integration tests
  format: JSON snapshots

  capture:
    - test_outputs
    - assertions
    - mock_data
    - expected_errors

  example:
    test: "user registration"
    baseline: "baselines/functional/user-registration-v1.json"
    includes:
      - success_response_format
      - validation_error_messages
      - database_state_after_registration
```

### Visual Baseline

```yaml
visual_baseline:
  description: UI screenshots for visual regression
  scope: Frontend components
  format: PNG images with metadata

  capture:
    - component_screenshots
    - page_screenshots
    - responsive_breakpoints
    - interaction_states

  example:
    component: "LoginForm"
    baseline: "baselines/visual/login-form-v2.png"
    metadata:
      viewport: 1920x1080
      theme: dark
      state: default
```

### Performance Baseline

```yaml
performance_baseline:
  description: Performance benchmarks
  scope: API response times, load tests
  format: JSON metrics

  capture:
    - response_times_p50_p95_p99
    - throughput_requests_per_second
    - resource_utilization
    - database_query_times

  example:
    endpoint: "/api/users"
    baseline: "baselines/performance/api-users-v1.json"
    metrics:
      p50_response_time_ms: 45
      p95_response_time_ms: 120
      p99_response_time_ms: 250
      throughput_rps: 1000
```

### API Contract Baseline

```yaml
api_baseline:
  description: API request/response contracts
  scope: REST/GraphQL APIs
  format: OpenAPI/JSON schemas

  capture:
    - request_schemas
    - response_schemas
    - status_codes
    - headers

  example:
    endpoint: "POST /api/auth/login"
    baseline: "baselines/api/auth-login-v3.yaml"
    contract:
      request_body_schema: LoginRequest
      success_response: 200 with token
      error_responses: [400, 401, 429]
```

## Baseline Manifest

```yaml
# .aiwg/testing/baselines/manifest.yaml

baselines:
  - id: functional-auth-v1
    type: functional
    created: 2026-01-28T10:00:00Z
    created_by: test-engineer
    git_commit: abc123def
    release: v1.2.0
    scope: Authentication flows
    files:
      - baselines/functional/login-v1.json
      - baselines/functional/logout-v1.json
      - baselines/functional/register-v1.json
    status: active
    approved_by: tech-lead
    notes: Initial baseline for auth module

  - id: visual-dashboard-v2
    type: visual
    created: 2026-01-20T14:30:00Z
    created_by: frontend-engineer
    git_commit: def456ghi
    release: v1.1.0
    scope: Dashboard UI components
    files:
      - baselines/visual/dashboard-desktop-v2.png
      - baselines/visual/dashboard-mobile-v2.png
    status: active
    approved_by: design-lead
    previous_baseline: visual-dashboard-v1
    changes: Updated color scheme per design system v2

  - id: performance-api-v1
    type: performance
    created: 2026-01-15T09:00:00Z
    created_by: devops-engineer
    git_commit: ghi789jkl
    release: v1.0.0
    scope: Core API endpoints
    files:
      - baselines/performance/api-benchmarks-v1.json
    status: active
    approved_by: architect
    notes: Pre-optimization baseline
```

## Baseline Comparison Report

```markdown
# Baseline Comparison Report

**Date**: 2026-01-28
**Baseline**: functional-auth-v1
**Current State**: HEAD (commit xyz789)
**Comparison Type**: Functional

## Executive Summary

**Status**: ⚠️ Drift Detected
**Changes**: 3 outputs differ from baseline
**Severity**: Medium - Unexpected behavior changes

| Metric | Baseline | Current | Drift |
|--------|----------|---------|-------|
| Tests Passing | 45/45 | 43/45 | -2 |
| Output Matches | 100% | 93.3% | -6.7% |
| New Failures | 0 | 2 | +2 |

## Detailed Comparison

### Test: user-login

**Status**: ⚠️ Drift

**Baseline** (functional-auth-v1):
```json
{
  "status": 200,
  "body": {
    "token": "jwt.header.payload.signature",
    "user": {
      "id": "uuid",
      "email": "[email protected]"
    }
  }
}
```

**Current**:
```json
{
  "status": 200,
  "body": {
    "token": "jwt.header.payload.signature",
    "user": {
      "id": "uuid",
      "email": "[email protected]",
      "name": "John Doe"  // NEW FIELD
    }
  }
}
```

**Analysis**: New `name` field added to response
**Impact**: Breaking change for clients expecting exact schema
**Recommendation**: Update baseline if intentional, or fix if bug

### Test: user-registration-invalid-email

**Status**: ❌ Failure

**Baseline**:
```json
{
  "status": 400,
  "body": {
    "error": "Invalid email format"
  }
}
```

**Current**:
```json
{
  "status": 500,
  "body": {
    "error": "Internal server error"
  }
}
```

**Analysis**: Validation now returns 500 instead of 400
**Impact**: Critical - Server error on invalid input
**Recommendation**: Fix immediately - regression in error handling

### Test: user-logout

**Status**: ✅ Match

Output matches baseline exactly. No drift detected.

## Drift Summary

| Test | Status | Drift Type | Severity |
|------|--------|------------|----------|
| user-login | ⚠️ Drift | Schema change | Medium |
| user-registration | ✅ Match | None | - |
| user-registration-invalid | ❌ Fail | Error code | High |
| user-logout | ✅ Match | None | - |
| token-refresh | ✅ Match | None | - |

## Root Cause Analysis

### Likely Causes

1. **user-login drift**: Intentional feature addition (commit abc123)
   - PR #789: "Add user name to login response"
   - Needs baseline update

2. **invalid-email failure**: Unintentional regression
   - Validation middleware broken
   - Returns 500 instead of 400
   - Introduced in commit def456

## Recommendations

### Immediate Actions

- [ ] Fix validation error handling (returns 500 instead of 400)
- [ ] Add test for proper error codes
- [ ] Verify error handling across all validation

### Baseline Updates

- [ ] Update baseline for user-login if name field is intentional
- [ ] Document breaking change in API changelog
- [ ] Notify API consumers of schema change

### Process Improvements

- [ ]