cloudflare-queues

What this skill does

# Cloudflare Queues

**Status**: Production Ready ✅ | **Last Verified**: 2025-12-27

**Dependencies**: cloudflare-worker-base (for Worker setup)

**Contents**: [Quick Start](#quick-start-10-minutes) • [Critical Rules](#critical-rules) • [Top Errors](#top-3-critical-errors) • [Use Cases](#common-use-cases) • [When to Load References](#when-to-load-references) • [Limits](#limits--quotas)

---

## Quick Start (10 Minutes)

### 1. Create Queue

```bash
bunx wrangler queues create my-queue
bunx wrangler queues list
```

### 2. Producer (Send Messages)

**wrangler.jsonc:**

```jsonc
{
  "name": "my-producer",
  "main": "src/index.ts",
  "queues": {
    "producers": [
      {
        "binding": "MY_QUEUE",
        "queue": "my-queue"
      }
    ]
  }
}
```

**src/index.ts:**

```typescript
import { Hono } from 'hono';

type Bindings = {
  MY_QUEUE: Queue;
};

const app = new Hono<{ Bindings: Bindings }>();

app.post('/send', async (c) => {
  await c.env.MY_QUEUE.send({
    userId: '123',
    action: 'process-order',
    timestamp: Date.now(),
  });

  return c.json({ status: 'queued' });
});

export default app;
```

### 3. Consumer (Process Messages)

**wrangler.jsonc:**

```jsonc
{
  "name": "my-consumer",
  "main": "src/consumer.ts",
  "queues": {
    "consumers": [
      {
        "queue": "my-queue",
        "max_batch_size": 10,
        "max_retries": 3,
        "dead_letter_queue": "my-dlq"
      }
    ]
  }
}
```

**src/consumer.ts:**

```typescript
import type { MessageBatch } from '@cloudflare/workers-types';

export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      console.log('Processing:', message.body);
      // Your logic here
    }
    // Implicit ack: returning successfully acknowledges all messages
  },
};
```

**Deploy:**

```bash
bunx wrangler deploy
```

**Load**: `references/setup-guide.md` for complete 6-step setup with DLQ configuration

---

## Critical Rules

### Always Do ✅

1. **Configure Dead Letter Queue** for production queues
2. **Use explicit ack()** for non-idempotent operations (DB writes, API calls)
3. **Validate message size** before sending (<128 KB)
4. **Use sendBatch()** for multiple messages (more efficient)
5. **Implement exponential backoff** for retries
6. **Set appropriate batch settings** based on workload
7. **Monitor queue backlog** and consumer errors
8. **Use ctx.waitUntil()** for async cleanup in consumers
9. **Handle errors gracefully** - log, alert, retry
10. **Let concurrency auto-scale** (don't set max_concurrency unless needed)

### Never Do ❌

1. **Never assume message ordering** - not guaranteed FIFO
2. **Never rely on implicit ack for non-idempotent ops** - use explicit ack()
3. **Never send messages >128 KB** - will fail
4. **Never delete queues with active messages** - data loss
5. **Never skip DLQ configuration** in production
6. **Never exceed 5000 msg/s per queue** - rate limit error
7. **Never process messages synchronously in loop** - use Promise.all()
8. **Never ignore message.attempts** - use for backoff logic
9. **Never set max_concurrency=1** unless you have a very specific reason
10. **Never forget to ack()** in explicit acknowledgement patterns

---

## Top 3 Critical Errors

### Error #1: Message Too Large

**Problem**: Message exceeds 128 KB limit

**Solution**: Store large data in R2, send reference

```typescript
// ❌ Wrong
await env.MY_QUEUE.send({ data: largeArray }); // >128 KB fails

// ✅ Correct
const message = { data: largeArray };
const size = new TextEncoder().encode(JSON.stringify(message)).length;

if (size > 128000) {
  const key = `messages/${crypto.randomUUID()}.json`;
  await env.MY_BUCKET.put(key, JSON.stringify(message));
  await env.MY_QUEUE.send({ type: 'large-message', r2Key: key });
} else {
  await env.MY_QUEUE.send(message);
}
```

### Error #2: Throughput Exceeded

**Problem**: Exceeding 5000 messages/second per queue

**Solution**: Use sendBatch() and rate limiting

```typescript
// ❌ Wrong
for (let i = 0; i < 10000; i++) {
  await env.MY_QUEUE.send({ id: i }); // Too fast!
}

// ✅ Correct
const messages = Array.from({ length: 10000 }, (_, i) => ({
  body: { id: i },
}));

// Send in batches of 100
for (let i = 0; i < messages.length; i += 100) {
  await env.MY_QUEUE.sendBatch(messages.slice(i, i + 100));
}
```

### Error #3: Entire Batch Retried When One Message Fails

**Problem**: Single message failure causes all messages to retry

**Solution**: Use explicit acknowledgement

```typescript
// ❌ Wrong - implicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
        message.body.id,
        message.body.amount
      ).run();
    }
    // If any fails, ALL retry!
  },
};

// ✅ Correct - explicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      try {
        await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
          message.body.id,
          message.body.amount
        ).run();

        message.ack(); // Only ack on success
      } catch (error) {
        console.error(`Failed: ${message.id}`, error);
        // Don't ack - will retry independently
      }
    }
  },
};
```

**Load `references/error-catalog.md` for all 10 errors including DLQ configuration, auto-scaling issues, message deletion prevention, and detailed solutions.**

---

## Common Use Cases

### Use Case 1: Basic Message Queue

**When**: Simple async job processing (emails, notifications)

**Quick Pattern**:

```typescript
// Producer
await env.MY_QUEUE.send({ type: 'email', to: '[email protected]' });

// Consumer (implicit ack - for idempotent operations)
export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      await sendEmail(message.body.to, message.body.content);
    }
  },
};
```

**Load**: `templates/queues-producer.ts` + `templates/queues-consumer-basic.ts`

### Use Case 2: Database Writes (Non-Idempotent)

**When**: Writing to database, must avoid duplicates

**Load**: `templates/queues-consumer-explicit-ack.ts` + `references/consumer-api.md`

### Use Case 3: Retry with Exponential Backoff

**When**: Calling rate-limited APIs, temporary failures

**Load**: `templates/queues-retry-with-delay.ts` + `references/error-catalog.md` (Error #2, #3)

### Use Case 4: Dead Letter Queue Pattern

**When**: Production systems, need to capture permanently failed messages

**Load**: `templates/queues-dlq-pattern.ts` + `references/setup-guide.md` (Step 4)

### Use Case 5: High Throughput Processing

**When**: Processing thousands of messages per second

**Quick Pattern**:

```jsonc
{
  "queues": {
    "consumers": [{
      "queue": "my-queue",
      "max_batch_size": 100,        // Large batches
      "max_batch_timeout": 5,       // Fast processing
      "max_concurrency": null       // Auto-scale
    }]
  }
}
```

**Load**: `references/best-practices.md` → Optimizing Throughput

---

## When to Load References

**Load `references/setup-guide.md` when**:
- User needs complete setup walkthrough (queue → producer → consumer → DLQ)
- First time setting up Cloudflare Queues
- Need production configuration examples
- Want complete end-to-end example

**Load `references/error-catalog.md` when**:
- Encountering any of the 10 documented errors
- Troubleshooting queue issues
- Messages not being delivered/processed
- Need prevention checklist

**Load `references/producer-api.md` when**:
- Need complete producer API reference
- Using send() or sendBatch() methods
- Need message format specifications
- Handling large messages or batches

**Load `references/consumer-api.md` when**:
- Need complete consumer API reference
- Using explicit ack(), retry(), or retryAll()
- Need batch processing patterns
- Implementing complex consumer logic

**Load `