elevenlabs-agents

What this skill does

# ElevenLabs Agents Platform

## Overview

ElevenLabs Agents Platform is a comprehensive solution for building production-ready conversational AI voice agents. The platform coordinates four core components:

1. **ASR (Automatic Speech Recognition)** - Converts speech to text (32+ languages, sub-second latency)
2. **LLM (Large Language Model)** - Reasoning and response generation (GPT, Claude, Gemini, custom models)
3. **TTS (Text-to-Speech)** - Converts text to speech (5000+ voices, 31 languages, low latency)
4. **Turn-Taking Model** - Proprietary model that handles conversation timing and interruptions

### 🚨 Package Updates (November 2025)

ElevenLabs migrated to new scoped packages in August 2025:

**DEPRECATED (Do not use):**
- `@11labs/react` → **DEPRECATED**
- `@11labs/client` → **DEPRECATED**

**Current packages:**
```bash
npm install @elevenlabs/[email protected]        # React SDK
npm install @elevenlabs/[email protected]       # JavaScript SDK
npm install @elevenlabs/[email protected] # React Native SDK
npm install @elevenlabs/[email protected] # Base SDK
npm install -g @elevenlabs/[email protected]  # CLI
```

If you have old packages installed, uninstall them first:
```bash
npm uninstall @11labs/react @11labs/client
```

### When to Use This Skill

Use this skill when:
- Building voice-enabled customer support agents
- Creating interactive voice response (IVR) systems
- Developing conversational AI applications
- Integrating telephony (Twilio, SIP trunking)
- Implementing voice chat in web/mobile apps
- Configuring agents via CLI ("agents as code")
- Setting up RAG/knowledge bases for agents
- Integrating MCP (Model Context Protocol) servers
- Building HIPAA/GDPR-compliant voice systems
- Optimizing LLM costs with caching strategies

### Platform Capabilities

**Design & Configure**:
- Multi-step workflows with visual builder
- System prompt engineering (6-component framework)
- 5000+ voices across 31 languages
- Pronunciation dictionaries (IPA/CMU formats)
- Speed control (0.7x-1.2x)
- RAG-powered knowledge bases
- Dynamic variables and personalization

**Connect & Deploy**:
- React SDK (`@elevenlabs/react`)
- JavaScript SDK (`@elevenlabs/client`)
- React Native SDK (`@elevenlabs/react-native`)
- Swift SDK (iOS/macOS)
- Embeddable widget
- Telephony integration (Twilio, SIP)
- Scribe (Real-Time Speech-to-Text) - Beta

**Operate & Optimize**:
- Automated testing (scenario, tool call, load)
- Conversation analysis and evaluation
- Analytics dashboard (resolution rates, sentiment, compliance)
- Privacy controls (GDPR, HIPAA, SOC 2)
- Cost optimization (LLM caching, model swapping, burst pricing)
- CLI for "agents as code" workflow

---

## 1. Quick Start (3 Integration Paths)

### Path A: React SDK (Embedded Voice Chat)

For building voice chat interfaces in React applications.

**Installation**:
```bash
npm install @elevenlabs/react zod
```

**Basic Example**:
```typescript
import { useConversation } from '@elevenlabs/react';
import { z } from 'zod';

export default function VoiceChat() {
  const { startConversation, stopConversation, status } = useConversation({
    // Public agent (no API key needed)
    agentId: 'your-agent-id',

    // OR private agent (requires API key)
    apiKey: process.env.NEXT_PUBLIC_ELEVENLABS_API_KEY,

    // OR signed URL (server-generated, most secure)
    signedUrl: '/api/elevenlabs/auth',

    // Client-side tools (browser functions)
    clientTools: {
      updateCart: {
        description: "Update the shopping cart",
        parameters: z.object({
          item: z.string(),
          quantity: z.number()
        }),
        handler: async ({ item, quantity }) => {
          console.log('Updating cart:', item, quantity);
          return { success: true };
        }
      }
    },

    // Event handlers
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
    onEvent: (event) => {
      switch (event.type) {
        case 'transcript':
          console.log('User said:', event.data.text);
          break;
        case 'agent_response':
          console.log('Agent replied:', event.data.text);
          break;
      }
    },

    // Regional compliance (GDPR, data residency)
    serverLocation: 'us' // 'us' | 'global' | 'eu-residency' | 'in-residency'
  });

  return (
    <div>
      <button onClick={startConversation}>Start Conversation</button>
      <button onClick={stopConversation}>Stop</button>
      <p>Status: {status}</p>
    </div>
  );
}
```

### Path B: CLI ("Agents as Code")

For managing agents via code with version control and CI/CD.

**Installation**:
```bash
npm install -g @elevenlabs/agents-cli
# or
pnpm install -g @elevenlabs/agents-cli
```

**Workflow**:
```bash
# 1. Authenticate
elevenlabs auth login

# 2. Initialize project (creates agents.json, tools.json, tests.json)
elevenlabs agents init

# 3. Create agent from template
elevenlabs agents add "Support Agent" --template customer-service

# 4. Configure in agent_configs/support-agent.json

# 5. Push to platform
elevenlabs agents push --env dev

# 6. Test
elevenlabs agents test "Support Agent"

# 7. Deploy to production
elevenlabs agents push --env prod
```

**Project Structure Created**:
```
your_project/
├── agents.json              # Agent registry
├── tools.json               # Tool configurations
├── tests.json               # Test configurations
├── agent_configs/           # Individual agent files
├── tool_configs/            # Tool configuration files
└── test_configs/            # Test configuration files
```

### Path C: API (Programmatic Agent Management)

For creating agents dynamically (multi-tenant, SaaS platforms).

**Installation**:
```bash
npm install elevenlabs
```

**Example**:
```typescript
import { ElevenLabsClient } from 'elevenlabs';

const client = new ElevenLabsClient({
  apiKey: process.env.ELEVENLABS_API_KEY
});

// Create agent
const agent = await client.agents.create({
  name: 'Support Bot',
  conversation_config: {
    agent: {
      prompt: {
        prompt: "You are a helpful customer support agent.",
        llm: "gpt-4o",
        temperature: 0.7
      },
      first_message: "Hello! How can I help you today?",
      language: "en"
    },
    tts: {
      model_id: "eleven_turbo_v2_5",
      voice_id: "your-voice-id"
    }
  }
});

console.log('Agent created:', agent.agent_id);
```

---

## 2. Agent Configuration

### System Prompt Architecture (6 Components)

ElevenLabs recommends structuring agent prompts using 6 components:

#### 1. Personality
Define the agent's identity, role, and character traits.

**Example**:
```
You are Alex, a friendly and knowledgeable customer support specialist at TechCorp.
You have 5 years of experience helping customers solve technical issues.
You're patient, empathetic, and always maintain a positive attitude.
```

#### 2. Environment
Describe the communication context (phone, web chat, video call).

**Example**:
```
You're speaking with customers over the phone. Communication is voice-only.
Customers may have background noise or poor connection quality.
Speak clearly and occasionally use thoughtful pauses for emphasis.
```

#### 3. Tone
Specify formality, speech patterns, humor, and verbosity.

**Example**:
```
Tone: Professional yet warm. Use contractions ("I'm" instead of "I am") to sound natural.
Avoid jargon unless the customer uses it first. Keep responses concise (2-3 sentences max).
Use encouraging phrases like "I'll be happy to help with that" and "Let's get this sorted for you."
```

#### 4. Goal
Define objectives and success criteria.

**Example**:
```
Primary Goal: Resolve customer technical issues on the first call.
Secondary Goals:
- Verify customer identity securely
- Document issue details accurately
- Offer proactive solutions
- End calls with confirmation that the issue is resolved

Success Criteria: Customer verbally confirms their issue is resolved.
```

#### 5. Guardrails
Set boundaries, prohibited topics,