celery-config-patterns

What this skill does

# celery-config-patterns

Provides production-ready Celery configuration templates for all major Python frameworks (Django, Flask, FastAPI, standalone) with complete broker setup (Redis, RabbitMQ), security best practices, and framework-specific integration patterns.

## Use When

- Configuring Celery for the first time in any Python project
- Integrating Celery with Django, Flask, FastAPI, or standalone applications
- Setting up Redis or RabbitMQ as message broker
- Creating production-ready Celery configurations
- Implementing task routing and queue configurations
- Setting up monitoring and logging for Celery
- Migrating Celery configurations between frameworks

## Directory Structure

```
celery-config-patterns/
├── SKILL.md                          # This file
├── scripts/
│   ├── validate-config.sh            # Validate Celery configuration
│   ├── detect-framework.sh           # Auto-detect Python framework
│   ├── generate-config.sh            # Generate framework-specific config
│   └── test-broker-connection.sh     # Test broker connectivity
├── templates/
│   ├── celery-app-standalone.py      # Standalone Celery app
│   ├── celery-app-django.py          # Django Celery integration
│   ├── celery-app-flask.py           # Flask Celery integration
│   ├── celery-app-fastapi.py         # FastAPI Celery integration
│   ├── config-redis.py               # Redis broker configuration
│   ├── config-rabbitmq.py            # RabbitMQ broker configuration
│   ├── tasks-example.py              # Sample task definitions
│   └── beat-schedule.py              # Celery Beat schedule config
└── examples/
    ├── django-setup.md               # Complete Django setup guide
    ├── flask-setup.md                # Complete Flask setup guide
    ├── fastapi-setup.md              # Complete FastAPI setup guide
    └── standalone-setup.md           # Standalone Python setup guide
```

## Instructions

### Step 1: Detect Framework

Use the detection script to identify the Python framework:

```bash
bash scripts/detect-framework.sh
```

**Detects:**
- Django (looks for `manage.py`, `settings.py`)
- Flask (looks for Flask imports, `app.py`)
- FastAPI (looks for FastAPI imports, `main.py`)
- Standalone (no framework detected)

### Step 2: Select Template

Based on framework detection, choose the appropriate template:

| Framework | Template | Integration File |
|-----------|----------|------------------|
| Django | `celery-app-django.py` | `projectname/celery.py` |
| Flask | `celery-app-flask.py` | `app/celery.py` or `celery.py` |
| FastAPI | `celery-app-fastapi.py` | `app/celery.py` or `celery.py` |
| Standalone | `celery-app-standalone.py` | `celery_app.py` |

### Step 3: Configure Broker

Choose broker configuration template:

**Redis** (Recommended for simplicity):
```python
# Use templates/config-redis.py
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
```

**RabbitMQ** (Recommended for production):
```python
# Use templates/config-rabbitmq.py
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672//'
CELERY_RESULT_BACKEND = 'rpc://'
```

### Step 4: Generate Configuration

Use the generation script:

```bash
bash scripts/generate-config.sh --framework=django --broker=redis
bash scripts/generate-config.sh --framework=flask --broker=rabbitmq
bash scripts/generate-config.sh --framework=fastapi --broker=redis
```

**Script will:**
1. Copy appropriate framework template
2. Apply broker configuration
3. Create environment file with placeholders
4. Add to .gitignore if needed
5. Generate setup documentation

### Step 5: Validate Configuration

Run validation script before starting Celery:

```bash
bash scripts/validate-config.sh
```

**Validates:**
- Celery app exists and is importable
- Broker connection is valid
- Configuration syntax is correct
- Required environment variables are set
- Task discovery paths are correct

### Step 6: Test Broker Connection

Verify broker connectivity:

```bash
bash scripts/test-broker-connection.sh
```

**Tests:**
- Broker URL is reachable
- Authentication credentials are valid
- Connection pool can be established
- Basic message routing works

## Template Descriptions

### celery-app-standalone.py

Standalone Celery application without web framework integration.

**Features:**
- Basic Celery app configuration
- Task autodiscovery from tasks.py
- Configurable broker and backend
- Logging setup
- Beat schedule integration

**Usage:**
```python
# celery_app.py
from celery import Celery

app = Celery('myapp')
app.config_from_object('celeryconfig')

@app.task
def add(x, y):
    return x + y
```

### celery-app-django.py

Django-specific Celery configuration using Django settings.

**Features:**
- Reads configuration from Django settings
- Auto-discovers tasks in all installed apps
- Uses Django's database for result backend (optional)
- Integrates with Django logging
- Supports Django-celery-beat for periodic tasks

**Usage:**
```python
# myproject/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
```

### celery-app-flask.py

Flask-specific Celery configuration with Flask app context.

**Features:**
- Flask app factory pattern support
- Celery app context integration
- Uses Flask configuration
- Compatible with Flask-Celery extensions
- Request context handling for tasks

**Usage:**
```python
# celery.py
from celery import Celery

def make_celery(app):
    celery = Celery(app.import_name)
    celery.conf.update(app.config)
    return celery

# app.py
celery = make_celery(app)
```

### celery-app-fastapi.py

FastAPI-specific Celery configuration with async support.

**Features:**
- FastAPI lifespan event integration
- Async task support
- Pydantic model validation in tasks
- Background task coordination
- API endpoint integration examples

**Usage:**
```python
# celery.py
from celery import Celery

celery_app = Celery('fastapi_app')
celery_app.config_from_object('celeryconfig')

# main.py
from fastapi import FastAPI
from celery_app import celery_app
```

### config-redis.py

Redis broker and result backend configuration.

**Features:**
- Connection pooling
- SSL/TLS support
- Sentinel configuration
- Health checks
- Connection retry logic

**Configuration:**
```python
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
CELERY_REDIS_MAX_CONNECTIONS = 50
```

### config-rabbitmq.py

RabbitMQ broker configuration with advanced routing.

**Features:**
- Virtual host configuration
- Exchange and queue declarations
- Routing keys and bindings
- Dead letter queues
- Priority queues

**Configuration:**
```python
CELERY_BROKER_URL = 'amqp://user:pass@localhost:5672//'
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_TASK_ROUTES = {
    'app.tasks.critical': {'queue': 'critical'},
    'app.tasks.normal': {'queue': 'default'},
}
```

### tasks-example.py

Example task definitions with best practices.

**Features:**
- Basic task examples
- Task with retry logic
- Task with rate limiting
- Task with time limits
- Task with custom routing
- Task with result expiration

**Examples:**
```python
@app.task(bind=True, max_retries=3)
def process_data(self, data):
    try:
        # Process data
        return result
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

@app.task(rate_limit='10/m')
def send_email(to, subject, body):
    # Send email
    pass
```

### beat-schedule.py

Celery Beat periodic task scheduling.

**Features:**
- Crontab schedules
- Interval schedules
- Solar schedules
- Task arguments and kwargs
- Timezone support

**Examples:**
```python
CELERY_BEAT_SCHEDULE = {
    'cleanup-every-midnight': {
        'task': 'app.tasks.cleanup',
        'schedule': crontab(hour=0, minute=0),
    },
    're