livekit-skills

What this skill does

# LiveKit Voice Agent Development

This skill provides guidance for building voice AI agents with the LiveKit Agents SDK. It covers both LiveKit Cloud and self-hosted deployments, using the `lk` CLI for documentation access and project management. All factual information about APIs, methods, and configurations must come from live documentation.


## MANDATORY: Read This Checklist Before Starting

Before writing ANY code, complete this checklist:

1. **Read this entire skill document** - Do not skip sections
2. **Set up LiveKit credentials** (Cloud project or self-hosted server) - You need `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`
3. **Set up documentation access** - Install `lk` CLI for `lk docs` commands
4. **Plan to write tests** - Every agent implementation MUST include tests (see testing section below)
5. **Verify all APIs against live docs** - Never rely on model memory for LiveKit APIs


## Setup

### LiveKit Cloud

LiveKit Cloud is the fastest way to get a voice agent running. It provides:
- Managed infrastructure (no servers to deploy)
- **LiveKit Inference** for AI models (no separate API keys needed)
- Built-in noise cancellation, turn detection, and other voice features
- Simple credential management

### Connect to Your Cloud Project

1. Sign up at [cloud.livekit.io](https://cloud.livekit.io) if you haven't already
2. Create a project (or use an existing one)
3. Get your credentials from the project settings:
   - `LIVEKIT_URL` - Your project's WebSocket URL (e.g., `wss://your-project.livekit.cloud`)
   - `LIVEKIT_API_KEY` - API key for authentication
   - `LIVEKIT_API_SECRET` - API secret for authentication

4. Set these as environment variables (typically in `.env.local`):
```bash
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-api-key
LIVEKIT_API_SECRET=your-api-secret
```

The LiveKit CLI can automate credential setup. Consult the CLI documentation for current commands.

### Use LiveKit Inference for AI Models

LiveKit Inference is one option for AI model access when using LiveKit Cloud. It provides access to leading AI model providers—all through your LiveKit credentials with no separate API keys needed.

Benefits of LiveKit Inference:
- No separate API keys to manage for each AI provider
- Billing consolidated through your LiveKit Cloud account
- Optimized for voice AI workloads

Consult the documentation for available models, supported providers, and current usage patterns. The documentation always has the most up-to-date information.

### Self-Hosted Setup

Self-hosting removes Cloud tier limits on deployments and concurrency. You control scaling directly.

#### Local development
Install and run the LiveKit server:
- macOS: `brew install livekit`
- Linux: `curl -sSL https://get.livekit.io | bash`

Start in dev mode:
```bash
livekit-server --dev
```
Default credentials: API key `devkey`, API secret `secret`.

Set environment variables:
```bash
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret
```

#### Production deployment
Deploy `livekit-server` via Docker, Kubernetes, or VMs on any provider (Hetzner, AWS, GCP, etc.). Consult `lk docs get-page /home/self-hosting` or see `references/self-hosting.md` for details. Agent servers run as regular processes managed by your infra tooling.

### Using Your Own Model Providers

When self-hosting or when you prefer your own API keys over LiveKit Inference, configure model providers directly via environment variables:

```bash
# STT (Speech-to-Text)
DEEPGRAM_API_KEY=your-key

# LLM
OPENAI_API_KEY=your-key

# TTS (Text-to-Speech)
ELEVEN_API_KEY=your-key
# or
CARTESIA_API_KEY=your-key
```

The Agents SDK has plugins for all major providers. Pass model identifiers directly:

**Node.js / TypeScript:**
```typescript
import { voice } from "@livekit/agents";

const session = new voice.AgentSession({
  stt: "deepgram/nova-3:multi",
  llm: "openai/gpt-4.1-mini",
  tts: "cartesia/sonic-3:voice-id",  // or "elevenlabs/..."
});
```

**Python:**
```python
session = AgentSession(
    stt="deepgram/nova-3",
    llm="openai/gpt-4.1-mini",
    tts="elevenlabs/...",  # or "cartesia/sonic-3:voice-id"
)
```

Consult `lk docs search "plugins"` for the full list of supported providers.

### Project Templates

Initialize a new agent project with the CLI:

**Backend agents:**
```bash
lk agent init my-agent --template agent-starter-python
lk agent init my-agent --template agent-starter-node
```

**Frontend apps (React/Next.js, React Native, Swift, Flutter, Android):**
```bash
lk agent init my-frontend --template agent-starter-react
lk agent init my-frontend --template agent-starter-react-native
```

Omit `--template` to see all available templates interactively.

## Critical Rule: Never Trust Model Memory for LiveKit APIs

LiveKit Agents is a fast-evolving SDK. Model training data is outdated the moment it's created. When working with LiveKit:

- **Never assume** API signatures, method names, or configuration options from memory
- **Never guess** SDK behavior or default values
- **Always verify** against live documentation before writing code
- **Always cite** the documentation source when implementing features

This rule applies even when confident about an API. Verify anyway.

## Use LiveKit CLI for Documentation

Before writing any LiveKit code, use the `lk docs` CLI commands for current, verified API information. This prevents reliance on stale model knowledge.

### Search documentation
```bash
lk docs search "voice agent quickstart"
lk docs search "handoffs and tasks"
```

### Fetch specific pages
```bash
lk docs get-page /agents/start/voice-ai-quickstart
lk docs get-page /agents/build/tools /agents/build/vision
```

### Search SDK source code
```bash
lk docs code-search "class AgentSession" --repo livekit/agents
lk docs code-search "@function_tool" --language Python --full-file
```

### Check changelogs
```bash
lk docs changelog livekit/agents
lk docs changelog pypi:livekit-agents --releases 5
lk docs changelog npm:livekit-agents --releases 5
```

### If CLI is not installed
Install the LiveKit CLI first:
- macOS: `brew install livekit-cli`
- Linux: `curl -sSL https://get.livekit.io/cli | bash`
- Windows: `winget install LiveKit.LiveKitCLI`

As a fallback, reference pages are available in the `references/` directory alongside this skill.

## Voice Agent Architecture Principles

Voice AI agents have fundamentally different requirements than text-based agents or traditional software. Internalize these principles:

### Latency Is Critical

Voice conversations are real-time. Users expect responses within hundreds of milliseconds, not seconds. Every architectural decision should consider latency impact:

- Minimize LLM context size to reduce inference time
- Avoid unnecessary tool calls during active conversation
- Prefer streaming responses over batch responses
- Design for the unhappy path (network delays, API timeouts)

### Context Bloat Kills Performance

Large system prompts and extensive tool lists directly increase latency. A voice agent with 50 tools and a 10,000-token system prompt will feel sluggish regardless of model speed.

Design agents with minimal viable context:
- Include only tools relevant to the current conversation phase
- Keep system prompts focused and concise
- Remove tools and context that aren't actively needed

### Users Don't Read, They Listen

Voice interface constraints differ from text:
- Long responses frustrate users—keep outputs concise
- Users cannot scroll back—ensure clarity on first delivery
- Interruptions are normal—design for graceful handling
- Silence feels broken—acknowledge processing when needed

## Workflow Architecture: Handoffs and Tasks

Complex voice agents should not be monolithic. LiveKit Agents supports structured workflows that maintain low latency while handling sophisticated use cases.

### The Problem with Monolithic Agents

A single agent handling an entire conversation flow accumulates:
- T