test-executor

What this skill does

# Test Executor Skill

## Purpose

Execute tests in adaptive loops, analyze test results, generate structured failure reports, and iterate with fixes until all tests pass. Works universally with any testing framework or project structure through intelligent adaptation and discovery.

## When to Use This Skill

Use this skill when:

- Test plan is ready (from `test-plan-generator` or manual creation)
- Ready to execute tests for implemented features
- Need to validate implementation correctness
- Want systematic test execution with detailed failure reporting
- Need to iterate on test failures with clear diagnostics
- Running tests as part of implementation validation

## Test Execution Workflow

### Phase 1: Preparation

1. **Read Test Plan**
   - Locate test-plan.md or equivalent
   - Identify tests to execute
   - Group tests by type (E2E, API, unit, integration, performance)
   - Determine execution order

2. **Discover Project Test Setup**
   - Identify testing framework
   - Find test commands
   - Locate test files
   - Check for test configuration

3. **Ensure Services are Running**
   - Identify required services (backend, frontend, database, etc.)
   - Check if services are running
   - Start services if needed
   - Verify service health

### Phase 2: Test Execution Loop

For each test in plan:

1. **Execute Test**
   - Run appropriate test command
   - Capture output (stdout, stderr)
   - Record execution time
   - Determine pass/fail status

2. **Analyze Results**
   - Parse test output
   - Extract error messages
   - Identify failure causes
   - Categorize failure type

3. **If Test Passes:**
   - Mark test as complete in plan: `- [x]`
   - Continue to next test

4. **If Test Fails:**
   - Generate failure report
   - Add to test-failures.md
   - Continue to next test (or stop if critical failure)

5. **Update Progress**
   - Track: X/Y tests passed
   - Update test plan with results

### Phase 3: Report Generation

1. **Create Test Failure Report**
   - Document failed tests
   - Include error messages and stack traces
   - Identify probable causes
   - Suggest fixes
   - Save as `test-failures.md`

2. **Summary Statistics**
   - Total tests run
   - Passed / Failed / Skipped
   - Execution time
   - Success rate

## Adaptive Test Discovery

### Discovering Test Framework

**Backend/Unit Tests:**
```bash
# Look for config files and dependencies
package.json → "jest", "mocha", "vitest"
requirements.txt → "pytest", "unittest"
*.csproj → MSTest, xUnit, NUnit
Cargo.toml → built-in Rust tests
go.mod → built-in Go tests
```

**Frontend/E2E Tests:**
```bash
# Look for E2E frameworks
package.json → "playwright", "cypress", "puppeteer"
Check for test directories: e2e/, tests/, __tests__/
```

**API Tests:**
```bash
# Look for API test patterns
*.http files (REST Client)
*.test.ts with fetch/axios calls
curl commands in scripts
Postman collections
```

### Discovering Test Commands

**Strategy:**
1. **Check package.json scripts:**
   ```json
   {
     "scripts": {
       "test": "jest",
       "test:e2e": "playwright test",
       "test:unit": "vitest run"
     }
   }
   ```

2. **Check Makefile:**
   ```makefile
   test:
       pytest tests/

   test-e2e:
       npm run test:e2e
   ```

3. **Check CI/CD config:**
   - `.github/workflows/*.yml`
   - `.gitlab-ci.yml`
   - `azure-pipelines.yml`

4. **Try common patterns:**
   ```bash
   npm test
   dotnet test
   pytest
   go test ./...
   cargo test
   make test
   ```

5. **Ask user if uncertain:**
   "How do you run tests in this project?"

## Test Types and Execution

### 1. E2E (End-to-End) Tests

**Characteristics:**
- Test complete user flows
- Require running frontend + backend
- Use browser automation (Playwright, Cypress, etc.)
- Typically slowest tests

**Execution with MCP Playwright:**
```bash
# If using Playwright via MCP
# Tests are executed through MCP Playwright tools
# Navigate, click, fill forms, assert results
```

**Execution with npm:**
```bash
npm run test:e2e
# or
npx playwright test
npx cypress run
```

**Services Required:**
- Frontend dev server (e.g., http://localhost:5174)
- Backend API server (e.g., http://localhost:5001)
- Database (e.g., PostgreSQL on :5432)

**Example Test from Plan:**
```markdown
- [ ] E2E: User can submit form and see confirmation
```

**Execution:**
1. Ensure all services running
2. Run E2E test command
3. Parse output for pass/fail
4. Capture screenshots if failed

### 2. API Tests

**Characteristics:**
- Test backend endpoints directly
- Don't require frontend
- Use HTTP requests (curl, httpie, fetch, etc.)
- Faster than E2E tests

**Execution with curl:**
```bash
# Example from test plan:
# "Test POST /api/forms creates form in database"

curl -X POST http://localhost:5001/api/forms \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"title":"Test Form","description":"Test"}'

# Check response status code
# Verify response body
# Query database to confirm creation
```

**Execution with test framework:**
```bash
# If project has API test suite
npm run test:api
dotnet test --filter Category=API
pytest tests/api/
```

**Services Required:**
- Backend API server
- Database

### 3. Unit Tests

**Characteristics:**
- Test individual functions/components
- No external dependencies
- Fast execution
- Largest quantity typically

**Execution:**
```bash
# JavaScript/TypeScript
npm test
npm run test:unit
jest
vitest run

# .NET
dotnet test
dotnet test --filter Category=Unit

# Python
pytest tests/unit/
python -m pytest

# Go
go test ./...

# Rust
cargo test
```

**Services Required:**
- None (unit tests are isolated)

### 4. Integration Tests

**Characteristics:**
- Test component interactions
- May require database or external services
- Medium execution speed

**Execution:**
```bash
# Similar to unit tests but may need services
dotnet test --filter Category=Integration
pytest tests/integration/
npm run test:integration
```

**Services Required:**
- Database (often)
- External APIs (sometimes)

### 5. Performance Tests

**Characteristics:**
- Test response time, throughput, resource usage
- Require production-like environment
- Generate metrics

**Execution:**
```bash
# Load testing tools
# ab (Apache Bench), wrk, k6, JMeter

# Example: 100 requests, 10 concurrent
ab -n 100 -c 10 http://localhost:5001/api/forms

# Parse output for:
# - Requests per second
# - Response times (mean, median, p95, p99)
# - Failures
```

## Service Management

### Starting Required Services

**Adaptive Service Detection:**

1. **Frontend:**
   ```bash
   # Detection: package.json with "dev" script
   # Start: npm run dev (background)
   # Health check: curl http://localhost:5174
   ```

2. **Backend:**
   ```bash
   # .NET: dotnet run (background)
   # Node: npm start or node server.js
   # Python: python app.py or flask run
   # Go: go run main.go
   ```

3. **Database:**
   ```bash
   # Detection: docker-compose.yml
   # Start: docker-compose up -d postgres
   # Health check: pg_isready or curl health endpoint
   ```

**Start Services Script (template in bundled resources):**
```bash
#!/bin/bash
# scripts/start_services.sh (customizable)

echo "Starting services..."

# Start database
docker-compose up -d postgres
sleep 2

# Start backend
cd backend && dotnet run &
BACKEND_PID=$!
sleep 5

# Start frontend
npm run dev &
FRONTEND_PID=$!
sleep 3

echo "Services started"
echo "Backend PID: $BACKEND_PID"
echo "Frontend PID: $FRONTEND_PID"
```

### Checking Service Health

```bash
# Backend health check
curl http://localhost:5001/health || echo "Backend not ready"

# Frontend health check
curl http://localhost:5174 || echo "Frontend not ready"

# Database health check
docker exec postgres pg_isready -U user -d db
```

## Test Output Parsing

### Parse Strategy

Different testing frameworks have different output formats. Parse adaptively:

**Jest/Vitest Output:**
```
PASS  tests/unit/EmailService.test.ts
  ✓ sends email successfully (45ms)
  ✓