production-deployment

What this skill does

# Production Deployment

Complete production deployment guide for ElevenLabs API integration including rate limiting patterns, comprehensive error handling strategies, monitoring setup, and testing frameworks.

## Overview

This skill provides battle-tested patterns for deploying ElevenLabs API integration to production environments with:

- **Rate Limiting**: Concurrency-aware rate limiting respecting plan limits
- **Error Handling**: Comprehensive error recovery and retry strategies
- **Monitoring**: Real-time metrics, logging, and alerting
- **Testing**: Load testing, concurrency validation, and production readiness checks

## Quick Start

### 1. Setup Monitoring Infrastructure

```bash
bash scripts/setup-monitoring.sh --project-name "my-elevenlabs-app" \
  --log-level "info" \
  --metrics-port 9090
```

This script:
- Configures Winston logging with rotation
- Sets up Prometheus metrics endpoints
- Creates health check endpoints
- Initializes error tracking

### 2. Deploy Production Configuration

```bash
bash scripts/deploy-production.sh --environment "production" \
  --api-key "$ELEVENLABS_API_KEY" \
  --concurrency-limit 10 \
  --region "us-east-1"
```

This script:
- Validates environment variables
- Applies rate limiting configuration
- Configures error handling middleware
- Sets up monitoring integrations
- Performs smoke tests

### 3. Test Rate Limiting

```bash
bash scripts/test-rate-limiting.sh --concurrency 20 \
  --duration 60 \
  --plan-tier "pro"
```

This script:
- Simulates concurrent requests
- Validates queue behavior
- Measures latency under load
- Generates performance report

## ElevenLabs Concurrency Limits

### Limits by Plan Tier

| Plan | Multilingual v2 | Turbo/Flash | STT | Music |
|------|-----------------|-------------|-----|-------|
| Free | 2 | 4 | 8 | N/A |
| Starter | 3 | 6 | 12 | 2 |
| Creator | 5 | 10 | 20 | 2 |
| Pro | 10 | 20 | 40 | 2 |
| Scale | 15 | 30 | 60 | 3 |
| Business | 15 | 30 | 60 | 3 |
| Enterprise | Elevated | Elevated | Elevated | Highest |

### Queue Management

When concurrency limits are exceeded:
- Requests are queued alongside lower-priority requests
- Typical latency increase: ~50ms
- Response headers include: `current-concurrent-requests`, `maximum-concurrent-requests`

### Real-World Capacity

A concurrency limit of 5 can typically support ~100 simultaneous audio broadcasts depending on:
- Audio generation speed
- User behavior patterns
- Request distribution

## Rate Limiting Patterns

### 1. Token Bucket Algorithm

Best for: Variable rate limiting with burst capacity

```javascript
// See templates/rate-limiter.js.template for full implementation
const limiter = new TokenBucketRateLimiter({
  capacity: 10,           // Max concurrent requests
  refillRate: 2,          // Tokens per second
  queueSize: 100          // Max queued requests
});
```

### 2. Sliding Window with Priority Queue

Best for: Enforcing strict concurrency limits with prioritization

```python
# See templates/error-handler.py.template for full implementation
limiter = SlidingWindowRateLimiter(
    max_concurrent=10
    window_size=60
    priority_levels=3
)
```

### 3. Adaptive Rate Limiting

Best for: Self-adjusting to API response headers

Monitors `current-concurrent-requests` and `maximum-concurrent-requests` headers to dynamically adjust rate limits.

## Error Handling Strategies

### Error Categories

**1. Rate Limit Errors (429)**
- Implement exponential backoff
- Queue requests for retry
- Monitor queue depth

**2. Service Errors (500-599)**
- Retry with exponential backoff
- Circuit breaker pattern
- Fallback to cached audio

**3. Client Errors (400-499)**
- Log for debugging
- Do not retry
- Return meaningful error to user

**4. Network Errors**
- Retry with linear backoff
- Timeout after 30 seconds
- Circuit breaker after 5 failures

### Circuit Breaker Pattern

```javascript
// Automatically opens circuit after threshold failures
const circuitBreaker = new CircuitBreaker({
  failureThreshold: 5
  resetTimeout: 60000
  monitorInterval: 5000
});
```

## Monitoring Setup

### Key Metrics to Track

**Request Metrics:**
- `elevenlabs_requests_total` - Total requests by status
- `elevenlabs_requests_duration_seconds` - Request latency histogram
- `elevenlabs_concurrent_requests` - Current concurrent requests
- `elevenlabs_queue_depth` - Queued requests waiting

**Error Metrics:**
- `elevenlabs_errors_total` - Total errors by type
- `elevenlabs_retries_total` - Total retry attempts
- `elevenlabs_circuit_breaker_state` - Circuit breaker state

**Business Metrics:**
- `elevenlabs_characters_generated` - Total characters processed
- `elevenlabs_audio_duration_seconds` - Total audio duration
- `elevenlabs_quota_used_percentage` - Quota utilization

### Logging Best Practices

**Structure logs with:**
- Request ID for tracing
- User ID for analysis
- Timestamp in ISO 8601
- Error stack traces
- Performance metrics

**Log Levels:**
- `error` - Failures requiring attention
- `warn` - Degraded performance, retries
- `info` - Request completion, key events
- `debug` - Detailed execution flow

### Alerting Rules

**Critical Alerts:**
- Error rate > 5% over 5 minutes
- Circuit breaker open for > 1 minute
- Queue depth > 500 requests

**Warning Alerts:**
- Latency p95 > 2 seconds
- Quota usage > 90%
- Retry rate > 20%

## Testing Frameworks

### Load Testing

Simulate production traffic patterns:

```bash
# Gradual ramp-up test
bash scripts/test-rate-limiting.sh \
  --pattern "ramp-up" \
  --start-rps 1 \
  --end-rps 10 \
  --duration 300
```

### Concurrency Validation

Verify concurrency limits are enforced:

```bash
# Burst test
bash scripts/test-rate-limiting.sh \
  --pattern "burst" \
  --concurrency 50 \
  --iterations 100
```

### Chaos Testing

Test error handling under adverse conditions:

```bash
# Simulate API failures
bash scripts/test-rate-limiting.sh \
  --pattern "chaos" \
  --failure-rate 0.1 \
  --duration 120
```

## Production Checklist

### Pre-Deployment

- [ ] Environment variables configured
- [ ] Rate limiting configured for plan tier
- [ ] Error handling middleware implemented
- [ ] Monitoring and logging configured
- [ ] Health check endpoints created
- [ ] Load testing completed
- [ ] Chaos testing completed

### Post-Deployment

- [ ] Smoke tests passed
- [ ] Metrics dashboard configured
- [ ] Alerts configured and tested
- [ ] On-call rotation established
- [ ] Runbooks documented
- [ ] Backup/fallback strategy tested

## Scripts

### setup-monitoring.sh

Configures comprehensive monitoring infrastructure:
- Winston logging with daily rotation
- Prometheus metrics exporter
- Health check endpoints
- Error tracking integration
- Custom metric collectors

**Usage:**
```bash
bash scripts/setup-monitoring.sh \
  --project-name "my-app" \
  --log-level "info" \
  --metrics-port 9090 \
  --health-port 8080
```

### deploy-production.sh

Production deployment orchestration:
- Environment validation
- Dependency installation
- Configuration deployment
- Service health checks
- Smoke test execution
- Rollback on failure

**Usage:**
```bash
bash scripts/deploy-production.sh \
  --environment "production" \
  --api-key "$ELEVENLABS_API_KEY" \
  --concurrency-limit 10 \
  --skip-tests false
```

### test-rate-limiting.sh

Comprehensive rate limiting test suite:
- Concurrency limit validation
- Queue behavior testing
- Latency measurement
- Error rate tracking
- Performance reporting

**Usage:**
```bash
bash scripts/test-rate-limiting.sh \
  --concurrency 20 \
  --duration 60 \
  --plan-tier "pro" \
  --pattern "ramp-up"
```

### validate-config.sh

Production configuration validator:
- Environment variable checks
- API key validation
- Rate limit configuration
- Monitoring setup verification
- Security audit

**Usage:**
```bash
bash scripts/validate-config.sh \
  --config-file "config/production.json" \
  --strict true
```

### rollback.sh

Automated rollback script:
- Reverts to previous deployment
- Rest