Workers AI

What this skill does

# Workers AI

## Purpose

This skill provides comprehensive guidance for using Workers AI, Cloudflare's AI inference platform. It covers available models, inference patterns, embedding generation, RAG (Retrieval Augmented Generation) architectures, AI Gateway integration, and best practices for AI workloads. Use this skill when implementing AI features, selecting models, building RAG systems, or optimizing AI inference on Workers.

## Workers AI Overview

Workers AI provides serverless AI inference at the edge with:
- **Text Generation**: LLMs for chat, completion, summarization
- **Embeddings**: Vector representations for semantic search
- **Image Generation**: Text-to-image models
- **Vision**: Image classification and object detection
- **Speech**: Text-to-speech and automatic speech recognition
- **Translation**: Language translation models

### Key Benefits

- **Edge deployment**: Low latency inference globally
- **No infrastructure**: Serverless, auto-scaling
- **Integrated**: Native integration with Workers, Vectorize, D1
- **Cost-effective**: Pay per inference, no minimum
- **Latest models**: Llama 3.1, Mistral, BAAI embeddings

## Model Categories

### Text Generation Models

**LLaMA 3.1 (Recommended)**:
- `@cf/meta/llama-3.1-8b-instruct` - Chat and instruction following
- Best for: Conversational AI, Q&A, summarization, general text generation
- Context window: 128K tokens
- Multilingual support

**Mistral**:
- `@cf/mistral/mistral-7b-instruct-v0.2` - Fast instruction following
- Best for: Quick responses, simpler tasks
- Context window: 32K tokens

**Qwen**:
- `@cf/qwen/qwen1.5-14b-chat-awq` - Quantized for efficiency
- Best for: Balance between speed and quality

See `references/workers-ai-models.md` for complete model catalog with specifications and use cases.

### Embedding Models

**BGE Base (Recommended for English)**:
- `@cf/baai/bge-base-en-v1.5` - High-quality English embeddings
- Dimensions: 768
- Best for: RAG, semantic search, English content

**BGE Large (Higher Quality)**:
- `@cf/baai/bge-large-en-v1.5` - Higher quality, more compute
- Dimensions: 1024
- Best for: When quality is critical

**BGE Small (Faster)**:
- `@cf/baai/bge-small-en-v1.5` - Faster, smaller model
- Dimensions: 384
- Best for: When speed is critical, large volumes

**Multilingual**:
- `@cf/baai/bge-m3` - Multilingual support
- Best for: Multi-language content

### Image Generation

**Stable Diffusion**:
- `@cf/stabilityai/stable-diffusion-xl-base-1.0` - Text-to-image
- `@cf/bytedance/stable-diffusion-xl-lightning` - Faster generation
- Best for: Creating images from text descriptions

### Vision Models

**Image Classification**:
- `@cf/microsoft/resnet-50` - Object recognition
- Best for: Classifying image content

### Speech Models

**Text-to-Speech**:
- `@cf/meta/m2m100-1.2b` - Multilingual speech synthesis

**Automatic Speech Recognition**:
- `@cf/openai/whisper` - Speech-to-text
- Best for: Transcribing audio

## Text Generation

### Basic Inference

```javascript
export default {
  async fetch(request, env, ctx) {
    const response = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
      messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: 'What is Cloudflare Workers?' }
      ]
    });

    return new Response(JSON.stringify(response));
  }
};
```

### Streaming Responses

```javascript
const stream = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
  messages: [
    { role: 'user', content: 'Write a story about...' }
  ],
  stream: true
});

return new Response(stream, {
  headers: { 'Content-Type': 'text/event-stream' }
});
```

### Model Parameters

```javascript
const response = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
  messages: [/* messages */],
  max_tokens: 512,        // Max tokens to generate
  temperature: 0.7,       // Creativity (0-1, higher = more random)
  top_p: 0.9,            // Nucleus sampling
  top_k: 40,             // Top-k sampling
  repetition_penalty: 1.2 // Penalize repetition
});
```

**Parameter guidelines**:
- **temperature**: 0.1-0.3 for factual, 0.7-0.9 for creative
- **max_tokens**: Set based on expected response length
- **top_p/top_k**: Usually leave at defaults unless fine-tuning behavior

## Embeddings

### Generating Embeddings

```javascript
const embeddings = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
  text: ['Hello world', 'Another sentence']
}) as { data: number[][] };

const vector1 = embeddings.data[0]; // [0.123, -0.456, ...]
const vector2 = embeddings.data[1];
```

**Important TypeScript note**: Always add `as { data: number[][] }` type assertion when using embeddings API.

### Batch Processing

```javascript
// Batch multiple texts for efficiency
const texts = documents.map(d => d.content);

// Process in batches of 100 (recommended batch size)
const batchSize = 100;
const allEmbeddings = [];

for (let i = 0; i < texts.length; i += batchSize) {
  const batch = texts.slice(i, i + batchSize);
  const result = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
    text: batch
  }) as { data: number[][] };

  allEmbeddings.push(...result.data);
}
```

### Text Chunking for Embeddings

For long documents, split into chunks before embedding:

```javascript
import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 500,      // Characters per chunk
  chunkOverlap: 50     // Overlap between chunks
});

const chunks = await splitter.splitText(longDocument);

// Generate embedding for each chunk
const embeddings = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
  text: chunks
}) as { data: number[][] };

// Store each chunk with its embedding
for (let i = 0; i < chunks.length; i++) {
  await env.VECTOR_INDEX.insert([{
    id: `${docId}-chunk-${i}`,
    values: embeddings.data[i],
    metadata: { text: chunks[i], docId, chunkIndex: i }
  }]);
}
```

See `references/rag-architecture-patterns.md` for complete RAG implementation patterns.

## RAG (Retrieval Augmented Generation)

### Basic RAG Pattern

```javascript
async function answerQuestion(question, env) {
  // 1. Generate question embedding
  const questionEmbedding = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
    text: [question]
  }) as { data: number[][] };

  // 2. Find similar documents
  const similar = await env.VECTOR_INDEX.query(questionEmbedding.data[0], {
    topK: 3,
    returnMetadata: true
  });

  // 3. Build context from retrieved documents
  const context = similar.matches
    .map(match => match.metadata.text)
    .join('\n\n');

  // 4. Generate answer with context
  const answer = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
    messages: [
      {
        role: 'system',
        content: 'Answer the question using only the provided context. If the answer is not in the context, say "I don\'t have enough information."'
      },
      {
        role: 'user',
        content: `Context:\n${context}\n\nQuestion: ${question}`
      }
    ]
  });

  return {
    answer: answer.response,
    sources: similar.matches.map(m => ({
      score: m.score,
      text: m.metadata.text
    }))
  };
}
```

### Advanced RAG with Reranking

```javascript
async function advancedRAG(question, env) {
  // 1. Retrieve more candidates (top 10)
  const questionEmbedding = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
    text: [question]
  }) as { data: number[][] };

  const candidates = await env.VECTOR_INDEX.query(questionEmbedding.data[0], {
    topK: 10
  });

  // 2. Rerank with LLM for relevance
  const reranked = [];
  for (const candidate of candidates.matches) {
    const relevance = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
      messages: [{
        role: 'user',
        content: `Rate the relevance of this passage to the question on a scale of 0-10:\n\nQuestion: ${question}\n\nPassage: ${candidate.metadata.text}\n\nRating (just the number):`
      }],
      max