elevenlabs-agents

What this skill does

# ElevenLabs Agents Platform

## Overview

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:

1. **ASR (Automatic Speech Recognition)** - Converts speech to text (32+ languages, sub-second latency)
2. **LLM (Large Language Model)** - Reasoning and response generation (GPT, Claude, Gemini, custom models)
3. **TTS (Text-to-Speech)** - Converts text to speech (5000+ voices, 31 languages, low latency)
4. **Turn-Taking Model** - Proprietary model that handles conversation timing and interruptions

### 🚨 Package Migration (August 2025)

**DEPRECATED (Do not use):** `@11labs/react` and `@11labs/client`

**Current packages:**
```bash
bun add @elevenlabs/[email protected]        # React SDK
bun add @elevenlabs/[email protected]       # JavaScript SDK
bun add @elevenlabs/[email protected]  # React Native SDK
bun add @elevenlabs/[email protected] # Base SDK
bun add -g @elevenlabs/[email protected]  # CLI
```

If migrating, uninstall old packages first: `npm uninstall @11labs/react @11labs/client`

---

## Quick Start

### Path A: React SDK (Embedded Voice Chat)

For building voice chat interfaces in React applications.

**Installation:**
```bash
bun add @elevenlabs/react zod
```

**Basic Example:**
```typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Authentication (choose one)
    agentId: 'your-agent-id',  // Public agent (no key needed)
    // apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,  // Private (dev only)
    // signedUrl: '/api/elevenlabs/auth',  // Signed URL (production)

    // Client-side tools
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onEvent: (event) => {
      if (event.type === 'transcript') console.log('User:', event.data.text);
      if (event.type === 'agent_response') console.log('Agent:', event.data.text);
    },

    // Regional compliance
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}
```

**Complete template:** See `templates/basic-react-agent.tsx`

### Path B: CLI ("Agents as Code")

For managing agents via code with version control and CI/CD. Load `references/cli-commands.md` when using CLI workflows.

**Quick workflow:**
```bash
bun add -g @elevenlabs/agents-cli
elevenlabs auth login
elevenlabs agents init
elevenlabs agents add "Support Agent" --template customer-service
# Edit agent_configs/support-agent.json
elevenlabs agents push --env dev
elevenlabs agents test "Support Agent"
```

**See:** `references/cli-commands.md` for complete CLI reference and workflows.

### Path C: API (Programmatic Agent Management)

For creating agents dynamically (multi-tenant, SaaS platforms). Load `references/api-reference.md` when using the API directly.

**Quick example:**
```typescript
import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: { prompt: "You are a helpful support agent.", llm: "gpt-4o" },
      first_message: "Hello! How can I help you today?"
    },
    tts: { model_id: "eleven_turbo_v2_5", voice_id: "your-voice-id" }
  }
});
```

**See:** `references/api-reference.md` for complete API reference.

---

## Agent Configuration

### System Prompt Framework

ElevenLabs recommends a 6-component prompt structure: **Personality** (identity/role), **Environment** (communication context), **Tone** (formality/speech patterns), **Goal** (objectives/success criteria), **Guardrails** (boundaries/ethics), and **Tools** (available functions).

**Example structure:**
```
Personality: You are Alex, a friendly customer support specialist at TechCorp.
Environment: Phone communication, voice-only, potential background noise.
Tone: Professional yet warm. Use contractions. Keep responses to 2-3 sentences.
Goal: Resolve issues on first call. Success = customer confirms resolution.
Guardrails: Never give medical/legal advice. Escalate if customer becomes abusive.
Tools: lookup_order(id), transfer_to_supervisor(), send_password_reset(email)
```

**Complete guide:** Load `references/system-prompt-guide.md` when configuring agent prompts or improving conversation quality.

### Turn-Taking Modes

| Mode | Behavior | Best For |
|------|----------|----------|
| **Eager** | Responds quickly, jumps in early | Fast-paced support, quick orders |
| **Normal** | Balanced, waits for natural breaks | General customer service (default) |
| **Patient** | Waits longer for detailed responses | Information collection, tutoring |

Configuration: `"turn": { "mode": "patient" }` in `conversation_config`

---

## Core Features Summary

### Voice & Language
- **Multi-Voice:** Dynamic voice switching (adds ~200ms latency per switch)
- **Pronunciation Dictionary:** IPA, CMU, or word substitutions for custom pronunciation
- **Speed Control:** 0.7x - 1.2x (1.0x = normal)
- **Languages:** 32+ languages with auto-detection and multi-language presets

### Knowledge Base (RAG)
Upload documents (PDF/TXT/DOCX) for semantic search during conversations. Agent retrieves relevant chunks automatically. Adds ~500ms latency per query. Documents must be indexed before use.

**See:** `references/api-reference.md` for knowledge base API and configuration.

### Tools (4 Types)
1. **Client Tools** - Execute in browser (UI updates, navigation, local storage)
2. **Server Tools (Webhooks)** - Execute on your backend (database, payments, CRM)
3. **MCP Tools** - Connect to Model Context Protocol servers (enterprise APIs)
4. **System Tools** - Built-in platform tools (end_conversation, transfer_call, mute_microphone, press_digit)

**See:** `references/tool-examples.md` when implementing tools. Load `references/api-reference.md` for webhook tool creation API.

---

## Top 3 Critical Errors

### Error 1: Package Deprecation (@11labs/*)

**Symptom:** Import errors, "module not found"

**Solution:**
```bash
npm uninstall @11labs/react @11labs/client
bun add @elevenlabs/[email protected] @elevenlabs/[email protected]

# Update imports:
import { useConversation } from '@elevenlabs/react';  // Not @11labs/react
```

### Error 2: Android Audio Cutoff (First Message)

**Symptom:** First agent message cuts off on Android only (iOS/web work fine)

**Solution:**
```typescript
const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  connectionDelay: {
    android: 3_000,  // 3 seconds for Android audio mode switch
    ios: 0,
    default: 0
  }
});
```

### Error 3: CSP (Content Security Policy) Violations

**Symptom:** "Refused to load the script..." errors, CSP blocks blob URLs

**Solution - Self-host worklet files:**
```bash
cp node_modules/@elevenlabs/client/dist/worklets/*.js public/elevenlabs/
```

```typescript
const { startConversation } = useConversation({
  agentId: 'your-agent-id',
  workletPaths: {
    'rawAudioProcessor': '/elevenlabs/rawAudioProcessor.worklet.js',
    'audioConcatProcessor': '/elevenlabs/audioConcatProcessor.worklet.js',
  }
});
```

**See all 17 errors:** Load `references/error-catalog.md` when troubleshooting errors, debugging webhook failures, RAG issues, or platform-specific problems.

---

## When to Load References

Load specific reference files