broker-configurations

What this skill does

# Broker Configurations

**Purpose:** Comprehensive message broker configuration for Celery with production-ready patterns for Redis, RabbitMQ, and Amazon SQS.

**Activation Triggers:**
- Broker connection errors
- Setting up new Celery project
- Implementing high availability
- SSL/TLS configuration needed
- Performance tuning required
- Multi-broker comparison
- Cloud deployment (AWS/GCP/Azure)
- Sentinel or cluster setup

**Key Resources:**
- `templates/redis-config.py` - Production Redis configuration
- `templates/rabbitmq-config.py` - RabbitMQ with quorum queues
- `templates/sqs-config.py` - AWS SQS with IAM roles
- `templates/connection-strings.env` - Connection string formats
- `templates/ssl-config.py` - SSL/TLS configuration
- `scripts/test-broker-connection.sh` - Connection testing
- `scripts/setup-redis.sh` - Redis installation and configuration
- `scripts/setup-rabbitmq.sh` - RabbitMQ cluster setup
- `examples/redis-sentinel.md` - High availability with Sentinel
- `examples/rabbitmq-ha.md` - RabbitMQ clustering and quorum queues
- `examples/sqs-setup.md` - Complete AWS SQS integration

## Broker Selection Guide

### Quick Comparison

| Feature | Redis | RabbitMQ | SQS |
|---------|-------|----------|-----|
| **Performance** | Excellent (small msgs) | Very Good | Good |
| **Reliability** | Good (with Sentinel) | Excellent (quorum) | Excellent |
| **Monitoring** | Yes | Yes | Limited |
| **Remote Control** | Yes | Yes | No |
| **Management** | Manual/Cloud | Manual | Fully Managed |
| **Cost** | Server/Cloud | Server/Cloud | Pay-per-use |
| **Best For** | Speed, simple setup | Reliability, features | AWS, serverless |

### Decision Matrix

**Choose Redis when:**
- Need maximum speed for small messages
- Want simple setup and maintenance
- Already using Redis for caching/results
- Budget for managed Redis (Upstash, ElastiCache)
- Can accept brief downtime during failover

**Choose RabbitMQ when:**
- Need guaranteed message delivery
- Require complex routing patterns
- Need worker remote control features
- Want built-in monitoring and management UI
- Have dedicated ops team for management

**Choose SQS when:**
- Running on AWS infrastructure
- Want zero operational overhead
- Need unlimited automatic scaling
- Prefer pay-per-use pricing
- Don't need worker remote control

## Redis Broker Setup

### 1. Basic Configuration

```bash
# Install dependencies
pip install "celery[redis]"

# Use template
cp templates/redis-config.py celeryconfig.py

# Configure environment
cp templates/connection-strings.env .env
# Edit .env with actual values
```

**Key settings:**
```python
# templates/redis-config.py
broker_url = 'redis://:password@localhost:6379/0'

broker_transport_options = {
    'visibility_timeout': 3600,
    'retry_on_timeout': True,
    'max_connections': 50,
}
```

### 2. Production Settings

**CRITICAL:** Set `maxmemory-policy noeviction` in Redis config:
```bash
# Run setup script
./scripts/setup-redis.sh --install --configure

# Or manually
redis-cli CONFIG SET maxmemory-policy noeviction
```

**Why:** Prevents Redis from evicting task data, which would cause task loss.

### 3. High Availability with Sentinel

```bash
# See comprehensive guide
cat examples/redis-sentinel.md

# Quick setup
docker-compose -f examples/docker-compose.sentinel.yml up -d

# Configure Celery for Sentinel
CELERY_BROKER_URL='sentinel://host1:26379;host2:26379;host3:26379/0'
```

**Connection string format:**
```python
# Sentinel URL
broker_url = 'sentinel://sentinel1:26379;sentinel2:26379/0'

broker_transport_options = {
    'master_name': 'mymaster',
    'sentinel_kwargs': {'password': 'sentinel_password'},
}
```

## RabbitMQ Broker Setup

### 1. Basic Configuration

```bash
# Install dependencies
pip install "celery[amqp]"

# Use template
cp templates/rabbitmq-config.py celeryconfig.py

# Run setup script
./scripts/setup-rabbitmq.sh --install --configure
```

**Key settings:**
```python
# templates/rabbitmq-config.py
broker_url = 'amqp://user:password@localhost:5672/vhost'

# CRITICAL for quorum queues
broker_transport_options = {
    'confirm_publish': True,  # Required!
}
```

### 2. Quorum Queues (Recommended)

**Provides:**
- Automatic replication
- Leader election on failure
- No message loss

```python
from kombu import Queue

task_queues = (
    Queue(
        'default',
        queue_arguments={
            'x-queue-type': 'quorum',
            'x-delivery-limit': 3,
        }
    ),
)
```

### 3. High Availability Cluster

```bash
# See comprehensive guide
cat examples/rabbitmq-ha.md

# Docker cluster setup
docker-compose -f examples/docker-compose.rabbitmq-cluster.yml up -d

# Verify cluster
docker exec rabbitmq-1 rabbitmqctl cluster_status
```

**HAProxy load balancing:**
```yaml
# Distribute connections across cluster
broker_url = 'amqp://user:password@haproxy:5670/vhost'
```

## AWS SQS Broker Setup

### 1. IAM Configuration

```bash
# Create IAM policy and user
aws iam create-policy \
  --policy-name CelerySQSPolicy \
  --policy-document file://examples/celery-sqs-policy.json

# Or use IAM role (recommended for EC2/ECS)
# See examples/sqs-setup.md for complete guide
```

### 2. Basic Configuration

```bash
# Install dependencies
pip install "celery[sqs]"

# Use template
cp templates/sqs-config.py celeryconfig.py
```

**With IAM role (recommended):**
```python
# No credentials needed
broker_url = 'sqs://'

broker_transport_options = {
    'region': 'us-east-1',
    'visibility_timeout': 3600,
    'polling_interval': 1,
    'wait_time_seconds': 10,  # Long polling
}
```

**With explicit credentials:**
```python
broker_url = 'sqs://access_key:secret_key@'
```

### 3. FIFO Queues

**For ordered task processing:**
```python
task_queues = (
    Queue(
        'celery-default.fifo',
        queue_arguments={
            'FifoQueue': 'true',
            'ContentBasedDeduplication': 'true',
        }
    ),
)

# Send with message group ID
task.apply_async(
    args=[data],
    properties={'MessageGroupId': 'user-123'}
)
```

### 4. Result Backend

**SQS doesn't support results - use S3 or DynamoDB:**
```python
# Option 1: S3
result_backend = 's3://my-bucket/celery-results/'

# Option 2: DynamoDB
result_backend = 'dynamodb://'
result_backend_transport_options = {
    'table_name': 'celery-results',
}

# Option 3: Redis (hybrid)
result_backend = 'redis://redis.example.com:6379/0'
```

## SSL/TLS Configuration

### 1. Redis with SSL

```python
# Use templates/ssl-config.py
from templates.ssl_config import app

# Or manually
broker_url = 'rediss://password@host:6380/0'  # Note: rediss://

broker_use_ssl = {
    'ssl_cert_reqs': ssl.CERT_REQUIRED,
    'ssl_ca_certs': '/path/to/ca.pem',
    'ssl_certfile': '/path/to/client-cert.pem',
    'ssl_keyfile': '/path/to/client-key.pem',
}
```

### 2. RabbitMQ with SSL

```python
broker_url = 'amqps://user:password@host:5671/vhost'  # Note: amqps://

broker_use_ssl = {
    'ssl_cert_reqs': ssl.CERT_REQUIRED,
    'ssl_ca_certs': '/path/to/ca.pem',
    'ssl_certfile': '/path/to/client-cert.pem',
    'ssl_keyfile': '/path/to/client-key.pem',
}
```

### 3. Environment Variables

```bash
# .env
BROKER_SSL_ENABLED=true
BROKER_SSL_CERT=/path/to/client-cert.pem
BROKER_SSL_KEY=/path/to/client-key.pem
BROKER_SSL_CA=/path/to/ca-cert.pem
BROKER_SSL_VERIFY_MODE=CERT_REQUIRED
```

## Testing and Validation

### 1. Test Connection

```bash
# Test any broker type
./scripts/test-broker-connection.sh redis
./scripts/test-broker-connection.sh rabbitmq
./scripts/test-broker-connection.sh sqs

# Check specific features
./scripts/test-broker-connection.sh redis --ssl
```

**Script checks:**
- Broker connectivity
- Authentication
- SSL/TLS validation
- Configuration correctness
- Performance baseline

### 2. Python Test

```python
from celery import Celery

app = Celery(broker='redis://localhost:6379/0')

# Test connection
try:
    with app.connection() as conn:
        conn.ensure_connection(max_retries=3, timeout=5)
        print("✅ Conne