deployment-pipeline

What this skill does

# Deployment Pipeline

## When to Use

Activate this skill when:
- Setting up or modifying CI/CD pipelines with GitHub Actions
- Deploying application changes to staging or production environments
- Planning environment promotion strategies (dev -> staging -> production)
- Implementing pre-deployment validation gates
- Configuring health checks and smoke tests for deployed services
- Planning or executing rollback procedures after a failed deployment
- Setting up canary or blue-green deployment strategies
- Troubleshooting deployment failures or pipeline errors

**Output:** Write deployment results to `deployment-report.md` with status, version deployed, health check results, and rollback instructions if needed.

Do NOT use this skill for:
- Building or optimizing Docker images (use `docker-best-practices`)
- Responding to production incidents (use `incident-response`)
- Setting up monitoring or alerting (use `monitoring-setup`)
- Infrastructure provisioning (Terraform, CloudFormation)

## Instructions

### Pipeline Stages Overview

Every deployment follows a strict four-stage pipeline. No stage may be skipped.

```
┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────────┐
│  BUILD   │───>│   TEST   │───>│ STAGING  │───>│  PRODUCTION  │
│          │    │          │    │          │    │              │
│ • Lint   │    │ • Unit   │    │ • Deploy │    │ • Canary 10% │
│ • Build  │    │ • Integ  │    │ • Smoke  │    │ • Monitor    │
│ • Image  │    │ • E2E    │    │ • QA     │    │ • Full 100%  │
└──────────┘    └──────────┘    └──────────┘    └──────────────┘
     Gate:           Gate:           Gate:            Gate:
  Build pass     Tests pass     Smoke pass      Health checks
  No lint err    Coverage ≥80%  Manual approve  Error rate <1%
```

### Stage 1: Build

Build stage validates code quality and produces deployable artifacts.

**Steps:**
1. **Lint and format check** -- Run `ruff check` and `ruff format --check` for Python, `eslint` and `prettier --check` for React
2. **Type check** -- Run `mypy` for Python, `tsc --noEmit` for TypeScript
3. **Build artifacts** -- Build Python wheel/sdist, build React production bundle
4. **Build Docker images** -- Tag with git SHA and branch name

**Gate criteria:** All checks pass, images build successfully.

```yaml
# GitHub Actions build stage
build:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Lint Python
      run: ruff check src/ && ruff format --check src/
    - name: Type check Python
      run: mypy src/
    - name: Build backend image
      run: docker build -t app-backend:${{ github.sha }} -f Dockerfile.backend .
    - name: Build frontend
      run: npm ci && npm run build
    - name: Build frontend image
      run: docker build -t app-frontend:${{ github.sha }} -f Dockerfile.frontend .
```

### Stage 2: Test

Run the full test suite. Never skip tests for "urgent" deployments.

**Steps:**
1. **Unit tests** -- `pytest tests/unit/ -v --cov=src --cov-report=xml`
2. **Integration tests** -- `pytest tests/integration/ -v` (requires test database)
3. **Frontend tests** -- `npm test -- --coverage`
4. **E2E tests** -- `npx playwright test` against a test environment
5. **Security scan** -- `pip-audit` for Python, `npm audit` for Node

**Gate criteria:** All tests pass, coverage >= 80%, no critical vulnerabilities.

```yaml
# GitHub Actions test stage
test:
  needs: build
  runs-on: ubuntu-latest
  services:
    postgres:
      image: postgres:16
      env:
        POSTGRES_DB: testdb
        POSTGRES_PASSWORD: testpass
      ports: ['5432:5432']
    redis:
      image: redis:7-alpine
      ports: ['6379:6379']
  steps:
    - uses: actions/checkout@v4
    - name: Run unit tests
      run: pytest tests/unit/ -v --cov=src --cov-report=xml
    - name: Run integration tests
      run: pytest tests/integration/ -v
      env:
        DATABASE_URL: postgresql://postgres:testpass@localhost:5432/testdb
    - name: Check coverage threshold
      run: coverage report --fail-under=80
```

### Stage 3: Staging Deployment

Deploy to staging environment for validation before production.

**Pre-deployment checklist:**
- [ ] All tests pass in CI
- [ ] Database migrations tested with `scripts/migration-dry-run.sh`
- [ ] Environment variables verified for staging
- [ ] Feature flags configured appropriately
- [ ] Dependent services verified available

**Steps:**
1. **Run migration dry-run** -- Validate Alembic migrations against staging DB clone
2. **Deploy to staging** -- Push images, apply migrations, restart services
3. **Run smoke tests** -- Execute `scripts/smoke-test.sh` against staging URL
4. **Run health checks** -- Execute `scripts/health-check.py` for all endpoints
5. **Manual QA** -- Team verifies critical user flows

**Gate criteria:** Smoke tests pass, health checks green, QA sign-off.

### Stage 4: Production Deployment

Production deployment uses canary strategy to minimize risk.

**Canary deployment steps:**
1. **Deploy canary (10% traffic)** -- Route 10% of traffic to new version
2. **Monitor for 10 minutes** -- Watch error rates, latency, resource usage
3. **Evaluate canary** -- If error rate < 1% and p99 latency within 20% of baseline, proceed
4. **Ramp to 50%** -- Increase traffic to 50%, monitor for 5 minutes
5. **Full rollout (100%)** -- Complete the deployment
6. **Post-deployment smoke tests** -- Run full smoke test suite

```
Canary Timeline:
  0 min    10 min   15 min   20 min
  |--------|--------|--------|
  10%      Check    50%      100%
  Deploy   Metrics  Ramp     Full
           OK?      Up       Rollout
           |
           No -> Rollback immediately
```

**Automatic rollback triggers:**
- Error rate exceeds 5% during canary
- p99 latency increases by more than 50%
- Health check failures on canary instances
- Memory usage exceeds 90% threshold

### Pre-Deployment Validation

Run these validations before any deployment. Use `scripts/deploy.sh --validate-only` for a dry run.

**Backend validation:**
```bash
# Verify migrations are consistent
alembic check

# Verify no pending migrations
alembic heads --verbose

# Test migration against staging clone
./skills/deployment-pipeline/scripts/migration-dry-run.sh \
  --db-url "$STAGING_DB_URL" \
  --output-dir ./deploy-validation/

# Verify all dependencies are pinned
pip-compile --dry-run requirements.in
```

**Frontend validation:**
```bash
# Verify build succeeds
npm run build

# Check bundle size limits
npx bundlesize

# Verify environment variables are set
node -e "const vars = ['REACT_APP_API_URL']; vars.forEach(v => { if(!process.env[v]) throw new Error(v + ' not set') })"
```

### Environment Promotion

Strict rules govern how changes move between environments.

| Aspect | Development | Staging | Production |
|--------|-------------|---------|------------|
| Deploy trigger | Push to `main` | Manual or auto after tests | Manual approval required |
| Database | Local PostgreSQL | Staging PostgreSQL | Production PostgreSQL (RDS) |
| Secrets | `.env` file | GitHub Secrets | AWS Secrets Manager |
| Log level | DEBUG | INFO | WARNING |
| Feature flags | All enabled | Per-feature | Gradual rollout |
| SSL | Self-signed | ACM cert | ACM cert |
| Replicas | 1 | 2 | 3+ (auto-scaled) |

**Promotion rules:**
1. Code must pass ALL gates in the previous stage
2. Database migrations must be backward-compatible (no column drops without migration window)
3. Environment variables must be configured BEFORE deployment
4. Feature flags must be set to correct state BEFORE deployment
5. Rollback plan must be documented BEFORE production deployment

### Health Checks

Every service exposes health check endpoints. The deployment pipeline validates these after every deployment.

**Required health check endpoints:**

```python
# FastAPI health check endpoints
@router.get("/health")
async def health():
    """Basic liveness check -- returns 200 if process is running."""
    return {"status": "healthy", "timestamp": datetime.utcno