deployment-configs

What this skill does

# Deployment Configurations

**Purpose:** Generate production-ready deployment configurations for Celery workers and beat schedulers across multiple deployment platforms.

**Activation Triggers:**
- Production deployment setup
- Docker/containerization requirements
- Kubernetes orchestration needs
- Systemd service configuration
- Health check implementation
- Graceful shutdown requirements
- Multi-worker scaling
- Environment-specific configuration

**Key Resources:**
- `scripts/deploy.sh` - Complete deployment orchestration
- `scripts/test-deployment.sh` - Validate deployment health
- `scripts/health-check.sh` - Comprehensive health verification
- `templates/docker-compose.yml` - Full Docker stack
- `templates/Dockerfile.worker` - Optimized worker container
- `templates/kubernetes/` - K8s manifests for workers and beat
- `templates/systemd/` - Systemd service units
- `templates/health-checks.py` - Python health check implementation
- `examples/` - Complete deployment scenarios

## Deployment Platforms

### Docker Compose (Development/Staging)

**Use Case:** Local development, staging environments, simple production setups

**Configuration:** `templates/docker-compose.yml`

**Services Included:**
- Redis (broker)
- PostgreSQL (result backend)
- Celery worker(s)
- Celery beat scheduler
- Flower monitoring
- Health check sidecar

**Quick Start:**
```bash
# Generate Docker configuration
./scripts/deploy.sh docker --env=staging

# Start all services
docker-compose up -d

# Scale workers
docker-compose up -d --scale celery-worker=4

# View logs
docker-compose logs -f celery-worker
```

**Key Features:**
- Multi-worker support with easy scaling
- Volume mounts for code hot-reload
- Environment-specific configurations
- Health checks with restart policies
- Networking between services
- Persistent data volumes

### Kubernetes (Production)

**Use Case:** Production environments requiring orchestration, auto-scaling, and high availability

**Manifests:** `templates/kubernetes/`

**Resources:**
- `celery-worker.yaml` - Worker Deployment with HPA
- `celery-beat.yaml` - Beat StatefulSet (singleton)
- `celery-configmap.yaml` - Environment configuration
- `celery-secrets.yaml` - Sensitive credentials
- `celery-service.yaml` - Internal service endpoints
- `celery-hpa.yaml` - Horizontal Pod Autoscaler

**Quick Start:**
```bash
# Generate K8s manifests
./scripts/deploy.sh kubernetes --namespace=production

# Apply configuration
kubectl apply -f kubernetes/

# Scale workers
kubectl scale deployment celery-worker --replicas=10

# Monitor status
kubectl get pods -l app=celery-worker
kubectl logs -f deployment/celery-worker
```

**Key Features:**
- Horizontal Pod Autoscaling based on CPU/queue depth
- ConfigMaps for environment variables
- Secrets management for credentials
- Rolling updates with zero downtime
- Resource limits and requests
- Liveness and readiness probes
- Pod disruption budgets
- Anti-affinity for worker distribution

### Systemd (Bare Metal/VMs)

**Use Case:** Traditional server deployments, VPS, dedicated servers

**Service Units:** `templates/systemd/`

**Services:**
- `celery-worker.service` - Worker daemon
- `celery-beat.service` - Beat scheduler daemon
- `celery-flower.service` - Monitoring dashboard

**Quick Start:**
```bash
# Generate systemd units
./scripts/deploy.sh systemd --workers=4

# Install services
sudo cp systemd/*.service /etc/systemd/system/
sudo systemctl daemon-reload

# Enable and start
sudo systemctl enable celery-worker@{1..4}.service celery-beat.service
sudo systemctl start celery-worker@{1..4}.service celery-beat.service

# Check status
sudo systemctl status celery-worker@*.service
sudo journalctl -u [email protected] -f
```

**Key Features:**
- Multi-instance worker support (@instance syntax)
- Automatic restart on failure
- Resource limits (CPU, memory)
- Graceful shutdown handling
- Log management via journald
- User/group isolation
- Environment file support

## Health Checks Implementation

### Python Health Check Module

**Location:** `templates/health-checks.py`

**Capabilities:**
- Broker connectivity verification
- Result backend validation
- Worker discovery and ping
- Queue depth monitoring
- Task execution test
- Memory and CPU metrics

**Usage:**
```python
from health_checks import CeleryHealthCheck

# Initialize checker
health = CeleryHealthCheck(app)

# Run all checks
status = health.run_all_checks()

# Individual checks
broker_ok = health.check_broker()
workers_ok = health.check_workers()
queues_ok = health.check_queue_depth(threshold=1000)
```

**Integration:**
```python
# Flask endpoint
@app.route('/health')
def health_check():
    checker = CeleryHealthCheck(celery_app)
    result = checker.run_all_checks()
    return jsonify(result), 200 if result['healthy'] else 503

# FastAPI endpoint
@app.get("/health")
async def health_check():
    checker = CeleryHealthCheck(celery_app)
    result = checker.run_all_checks()
    return result if result['healthy'] else JSONResponse(
        status_code=503, content=result
    )
```

### Shell Script Health Checks

**Location:** `scripts/health-check.sh`

**Features:**
- Standalone health verification
- Exit code compatibility (0=healthy, 1=unhealthy)
- JSON output for parsing
- Configurable timeouts
- Retry logic

**Usage:**
```bash
# Basic health check
./scripts/health-check.sh

# With custom timeout
./scripts/health-check.sh --timeout=30

# JSON output
./scripts/health-check.sh --json

# Specific checks
./scripts/health-check.sh --check=broker,workers
```

**Docker Integration:**
```dockerfile
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD /app/scripts/health-check.sh || exit 1
```

**Kubernetes Integration:**
```yaml
livenessProbe:
  exec:
    command: ["/app/scripts/health-check.sh"]
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  exec:
    command: ["/app/scripts/health-check.sh", "--check=workers"]
  initialDelaySeconds: 10
  periodSeconds: 5
```

## Deployment Scripts

### Main Deployment Orchestrator

**Script:** `scripts/deploy.sh`

**Capabilities:**
- Platform detection and configuration
- Environment-specific settings
- Secret management validation
- Pre-deployment checks
- Configuration generation
- Deployment execution
- Post-deployment verification

**Usage:**
```bash
# Deploy to Docker
./scripts/deploy.sh docker --env=production

# Deploy to Kubernetes
./scripts/deploy.sh kubernetes --namespace=prod --replicas=10

# Deploy systemd services
./scripts/deploy.sh systemd --workers=4 --user=celery

# Dry run (generate configs only)
./scripts/deploy.sh kubernetes --dry-run

# With custom configuration
./scripts/deploy.sh docker --config=custom-config.yml
```

**Pre-deployment Checks:**
- Broker accessibility
- Result backend connectivity
- Required environment variables
- Secret availability (no hardcoded keys)
- Python dependencies
- Celery app importability
- Task discovery

### Deployment Testing

**Script:** `scripts/test-deployment.sh`

**Test Coverage:**
- Service availability
- Health endpoint responses
- Worker registration
- Task execution end-to-end
- Beat schedule validation
- Monitoring dashboard access
- Log output verification
- Resource utilization

**Usage:**
```bash
# Test Docker deployment
./scripts/test-deployment.sh docker

# Test Kubernetes deployment
./scripts/test-deployment.sh kubernetes --namespace=prod

# Test systemd services
./scripts/test-deployment.sh systemd

# Verbose output
./scripts/test-deployment.sh docker --verbose

# Continuous monitoring
./scripts/test-deployment.sh docker --watch --interval=60
```

**Test Scenarios:**
1. Submit test task to each queue
2. Verify task completion
3. Check worker logs for errors
4. Validate beat schedule execution
5. Test graceful shutdown
6. Verify task retry behavior
7. Check monitoring metrics

## Configuration Templates

### Docker Compose Template

**File:** `templates/docker-compose.yml`

**Highlights:**
- Multi-stage build support
- Environment v