regression-report

What this skill does

# regression-report

Generate comprehensive regression analysis reports combining bisect, baseline, and metrics data.

## Triggers


Alternate expressions and non-obvious activations (primary phrases are matched automatically from the skill description):

- "full regression report" → comprehensive regression analysis
- "test health" → regression summary report

## Purpose

This skill produces comprehensive regression reports by:
- Synthesizing data from bisect, baseline, and metrics
- Correlating regressions with code changes
- Identifying systemic issues and patterns
- Providing actionable recommendations
- Creating executive summaries
- Supporting postmortem analysis

## Behavior

When triggered, this skill:

1. **Gathers regression data**:
   - Load bisect reports for root cause
   - Import baseline comparison results
   - Fetch regression metrics and trends
   - Retrieve issue tracker data
   - Collect CI/CD logs

2. **Correlates information**:
   - Link regressions to code changes
   - Map to requirements and features
   - Identify common root causes
   - Connect to team/component ownership

3. **Analyzes impact**:
   - Assess user impact
   - Calculate downtime/degradation
   - Measure business cost
   - Evaluate reputation impact

4. **Identifies patterns**:
   - Recurring regression types
   - High-risk components
   - Time-based patterns
   - Process gaps

5. **Generates insights**:
   - Root cause categories
   - Prevention strategies
   - Process improvements
   - Tooling recommendations

6. **Produces reports**:
   - Executive summary
   - Technical deep-dive
   - Action plan
   - Lessons learned

## Report Types

### Incident Report

```yaml
incident_report:
  description: Single regression deep-dive
  scope: One specific regression incident
  audience: Technical team and stakeholders

  sections:
    - incident_summary
    - timeline
    - root_cause_analysis
    - impact_assessment
    - resolution_steps
    - prevention_measures
    - lessons_learned
```

### Sprint Regression Report

```yaml
sprint_report:
  description: All regressions in a sprint
  scope: Sprint boundary
  audience: Development team

  sections:
    - sprint_summary
    - regression_list
    - metrics_summary
    - hotspot_analysis
    - recommendations
    - goals_for_next_sprint
```

### Release Regression Report

```yaml
release_report:
  description: Regression analysis for release
  scope: Release cycle
  audience: Release management and stakeholders

  sections:
    - release_summary
    - regression_timeline
    - escape_analysis
    - quality_gates_review
    - lessons_learned
    - release_readiness
```

### Quarterly Regression Analysis

```yaml
quarterly_report:
  description: Strategic regression analysis
  scope: 3 months
  audience: Leadership and engineering managers

  sections:
    - executive_summary
    - trend_analysis
    - strategic_insights
    - investment_recommendations
    - process_improvements
    - success_metrics
```

## Comprehensive Report Format

```markdown
# Comprehensive Regression Report

**Period**: Sprint 13 (2026-01-14 to 2026-01-28)
**Project**: User Service
**Report Type**: Sprint Regression Analysis
**Generated**: 2026-01-28 17:00:00
**Analyzer**: regression-report skill

---

## Executive Summary

**Overall Assessment**: ⚠️ Acceptable Quality with Concerns

| Metric | Value | Status |
|--------|-------|--------|
| Regressions Detected | 4 | ✅ Within target (< 5) |
| Production Escapes | 1 | ⚠️ Above target (0 preferred) |
| Mean Time to Detect | 8.5h | ✅ Excellent |
| Mean Time to Fix | 18.7h | ⚠️ Close to target |
| User Impact | 500 users | ⚠️ Medium |

**Key Findings**:
- Regression rate within acceptable range but auth module concerning
- One production escape indicates staging test gap
- Detection speed excellent, fix time acceptable
- Systemic issue: Integration testing coverage insufficient

**Priority Actions**:
1. Add integration tests for authentication flows
2. Improve staging environment test coverage
3. Implement regression test requirement for all fixes

---

## Regression Inventory

### REG-001: JWT Issuer Validation Breaks Existing Sessions

**Severity**: High
**Status**: Fixed (deployed 2026-01-16)
**Detection**: Automated test failure
**Environment**: Staging

**Timeline**:
- 2026-01-15 14:32: Breaking commit merged (pqr901)
- 2026-01-15 16:15: CI test failure detected (2h MTTD)
- 2026-01-15 18:45: Root cause identified via bisect
- 2026-01-16 09:30: Fix deployed (15h MTTF)

**Root Cause**: Required `iss` claim in JWT validation without backward compatibility

**Impact**: All existing user sessions invalidated

**Files Changed**:
- src/auth/validate-token.ts (+15/-5)
- src/auth/generate-token.ts (+8/-2)

**Fix**: Added feature flag for gradual rollout

**Prevention**:
- [ ] Add integration tests for auth changes
- [ ] Require backward compatibility review

**References**:
- Bisect Report: @.aiwg/testing/regression-bisect-jwt.md
- Issue: #456
- PR: #789

---

### REG-002: User Profile Update Returns 500 on Invalid Email

**Severity**: Critical
**Status**: Fixed (deployed 2026-01-18)
**Detection**: Production monitoring
**Environment**: Production

**Timeline**:
- 2026-01-17 08:00: Deployed to production
- 2026-01-17 20:15: Error spike detected (12h MTTD)
- 2026-01-18 02:30: Root cause identified
- 2026-01-18 04:15: Hotfix deployed (8h MTTF)

**Root Cause**: Validation middleware error handling broken, returns 500 instead of 400

**Impact**:
- 500 users affected
- 45 support tickets
- 12 hours partial service degradation

**Files Changed**:
- src/api/validation-middleware.ts (+3/-8)

**Cost**:
- Engineering: 8 hours incident response
- Support: 12 hours ticket triage
- Reputation: Customer satisfaction impact

**Fix**: Restored proper error handling

**Prevention**:
- [ ] Add error code validation tests
- [ ] Improve staging test coverage
- [ ] Add monitoring for error rate spikes

**References**:
- Incident Report: @.aiwg/incidents/INC-2026-001.md
- Issue: #478
- Hotfix PR: #791

---

### REG-003: Password Reset Email Fails for Gmail Users

**Severity**: Medium
**Status**: Fixed (deployed 2026-01-22)
**Detection**: Manual QA testing
**Environment**: Staging

**Timeline**:
- 2026-01-20 11:00: Code merged
- 2026-01-21 15:30: QA detected (28.5h MTTD)
- 2026-01-22 10:00: Fix deployed (18.5h MTTF)

**Root Cause**: Email template contained invalid HTML characters rejected by Gmail

**Impact**: Password reset broken for ~40% of user base (Gmail users)

**Files Changed**:
- templates/email/password-reset.html (+5/-3)

**Fix**: HTML entity encoding for special characters

**Prevention**:
- [ ] Add email template validation
- [ ] Test with multiple email providers
- [ ] Add email delivery monitoring

**References**:
- Issue: #482
- PR: #794

---

### REG-004: Dashboard Widget Load Time Increased

**Severity**: Low
**Status**: Fixed (deployed 2026-01-26)
**Detection**: Performance baseline comparison
**Environment**: Staging

**Timeline**:
- 2026-01-24 09:00: Performance test detected slowdown
- 2026-01-24 10:15: Root cause identified (1.25h MTTD)
- 2026-01-26 08:00: Optimization deployed (45.75h MTTF)

**Root Cause**: N+1 query introduced in dashboard widget

**Impact**: Dashboard load time increased from 0.8s to 2.3s

**Files Changed**:
- src/dashboard/widgets/activity-feed.ts (+12/-5)

**Fix**: Added eager loading to eliminate N+1

**Prevention**:
- [ ] Add performance regression tests
- [ ] Code review focus on query optimization

**References**:
- Baseline Comparison: @.aiwg/testing/baseline-comparison-perf.md
- Issue: #485
- PR: #797

---

## Impact Analysis

### User Impact

| Regression | Users Affected | Duration | Severity |
|------------|----------------|----------|----------|
| REG-001 | 0 (staging) | N/A | High (avoided) |
| REG-002 | 500 | 12h | Critical |
| REG-003 | 0 (staging) | N/A | Medium (avoided) |
| REG-004 | 0 (staging) | N/A | Low (avoided) |

**Total Production Impact**: 500 users, 12 h