tts-livekit-plugin

What this skill does

# Self-Hosted TTS API and LiveKit Plugin

Build production-ready self-hosted Text-to-Speech APIs using HuggingFace models and integrate them with LiveKit voice agents through custom plugins.

---

## Overview

This skill enables you to:
- **Build TTS APIs** using state-of-the-art HuggingFace models (Parler-TTS, F5-TTS, XTTS-v2)
- **Create LiveKit plugins** that connect voice agents to your self-hosted TTS
- **Implement streaming** for low-latency real-time synthesis
- **Deploy to production** with Docker, Kubernetes, or cloud platforms

**When to use this skill:**
- Creating cost-effective voice agents without cloud TTS fees
- Requiring custom voice characteristics or multilingual support
- Building privacy-focused systems with on-premise TTS
- Developing voice agents that need streaming audio synthesis

---

## Implementation Process

### Step 1: Choose Your TTS Model

Select the best model for your use case from the HuggingFace ecosystem.

**Load model comparison:** [TTS Models Reference](./references/tts-models.md)

**Quick Selection Guide:**

| Use Case | Recommended Model | Why |
|----------|------------------|-----|
| Production voice agents | Parler-TTS Mini | Fast, CPU-friendly, text-based voice control |
| High-quality synthesis | Parler-TTS Large / F5-TTS | Superior natural quality |
| Multilingual support | XTTS-v2 | 17+ languages, voice cloning |
| Cost optimization | Parler-TTS Mini on CPU | Runs efficiently without GPU |

**Example decision:**
- User needs: "Fast, conversational voice for customer support agent"
- Model: `parler-tts/parler-tts-mini-v1`
- Device: CPU (for cost) or GPU (for speed)
- Voice description: `"A friendly, professional voice with moderate pace"`

---

### Step 2: Build the TTS API Server

Create a FastAPI server that hosts the TTS model with both batch and streaming endpoints.

**Use the provided implementation:**

The skill includes a complete TTS API server at `tts-api/main.py` that supports:
- **Batch synthesis**: POST `/synthesize` for simple text-to-speech
- **Streaming synthesis**: WebSocket `/ws/synthesize` for real-time incremental synthesis
- **Multiple models**: Parler-TTS, F5-TTS, XTTS-v2 (configurable)
- **Optimizations**: Model caching, async synthesis, sentence-level streaming

**Quick start:**

```bash
# Navigate to TTS API directory
cd tts-api

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env to set:
#   TTS_MODEL_TYPE=parler
#   TTS_MODEL_NAME=parler-tts/parler-tts-mini-v1
#   TTS_DEVICE=cpu

# Run the server
python main.py
```

The server will:
1. Load the model on startup (may take 1-2 minutes)
2. Listen on port 8001
3. Provide health check at `GET /health`
4. Accept synthesis requests at `POST /synthesize` and `WS /ws/synthesize`

**Test the API:**

```bash
# Test batch synthesis
curl -X POST http://localhost:8001/synthesize \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello! This is a test.", "format": "wav"}' \
  --output test.wav

# Check health
curl http://localhost:8001/health
```

**For detailed implementation patterns and best practices:**
- [API Implementation Guide](./references/api-implementation.md)

**Key implementation details:**

The provided `tts-api/main.py` includes:
- **Model loading**: Singleton pattern with startup event
- **Sentence-level streaming**: Splits text at sentence boundaries for natural prosody
- **Keepalive messages**: Prevents WebSocket timeouts (5s interval)
- **End-of-stream signaling**: Explicit completion messages
- **Error handling**: Graceful failures with detailed error messages
- **Audio formats**: PCM int16 (streaming), WAV, MP3 (batch)

---

### Step 3: Create the LiveKit TTS Plugin

Build a LiveKit plugin that connects your voice agents to the self-hosted TTS API.

**Use the provided plugin implementation:**

The skill includes a complete LiveKit plugin at `livekit-plugin-custom-tts/` with:
- **TTS class**: Main plugin interface
- **ChunkedStream**: Streaming synthesis session
- **WebSocket communication**: Bi-directional streaming
- **Examples**: Voice agent integration and standalone usage

**Install the plugin:**

```bash
# Navigate to plugin directory
cd livekit-plugin-custom-tts

# Install in development mode
pip install -e .

# Or install from source
pip install .
```

**Use in a voice agent:**

```python
from livekit import agents
from livekit.agents import AgentSession
from livekit.plugins import openai, deepgram, silero
from livekit.plugins import custom_tts

async def entrypoint(ctx: agents.JobContext):
    # Initialize session with custom TTS
    session = AgentSession(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-2-general"),
        llm=openai.LLM(model="gpt-4o-mini"),
        # Use custom self-hosted TTS
        tts=custom_tts.TTS(
            api_url="http://localhost:8001",
            options=custom_tts.TTSOptions(
                voice_description="A friendly, conversational voice.",
                sample_rate=24000,
            ),
        ),
    )

    await ctx.connect()
    await session.start(agent=YourAgent(), room=ctx.room)
```

**For detailed plugin development patterns:**
- [Plugin Development Guide](./references/plugin-development.md)

**Key plugin features:**

The provided implementation includes:
- **Streaming synthesis**: Iterates over audio chunks as they're generated
- **Keepalive**: Maintains long-running WebSocket connections
- **Error recovery**: Graceful handling of connection failures
- **Resource cleanup**: Proper task cancellation and WebSocket closure
- **LiveKit integration**: Follows LiveKit TTS plugin interface exactly

---

### Step 4: Test the Integration

Verify that the TTS API and plugin work together correctly.

**Testing levels:**

**1. API-level testing:**

```bash
# Start the TTS API
cd tts-api
python main.py

# In another terminal, test synthesis
curl -X POST http://localhost:8001/synthesize \
  -H "Content-Type: application/json" \
  -d '{"text": "Testing TTS API", "format": "wav"}' \
  --output test.wav

# Play the audio
ffplay test.wav  # or open test.wav
```

**2. Plugin-level testing:**

Use the provided example script:

```bash
cd livekit-plugin-custom-tts/examples
python basic_usage.py
```

This will:
- Connect to the TTS API
- Synthesize test text
- Save audio to `output.wav`

**3. Voice agent testing:**

Use the provided voice agent example:

```bash
# Set environment variables
export LIVEKIT_URL="wss://your-livekit.cloud"
export LIVEKIT_API_KEY="your-api-key"
export LIVEKIT_API_SECRET="your-api-secret"
export OPENAI_API_KEY="your-openai-key"
export DEEPGRAM_API_KEY="your-deepgram-key"

# Run the voice agent
cd livekit-plugin-custom-tts/examples
python voice_agent.py start
```

**Verification checklist:**

- [ ] TTS API health endpoint returns OK
- [ ] Batch synthesis produces audio files
- [ ] WebSocket streaming works without disconnections
- [ ] Plugin synthesizes text successfully
- [ ] Voice agent speaks using custom TTS
- [ ] Audio quality is acceptable
- [ ] Latency is reasonable (<1 second for short sentences)

---

### Step 5: Deploy to Production

Deploy the TTS API and voice agent to production infrastructure.

**Deployment options:**

**Option 1: Docker Compose (Quick Start)**

Use the provided configuration:

```bash
# Create deployment directory
mkdir deployment
cd deployment

# Copy TTS API
cp -r ../tts-api .

# Set environment variables
export LIVEKIT_URL="wss://your-livekit.cloud"
export LIVEKIT_API_KEY="your-api-key"
# ... other vars

# Run docker-compose
docker-compose up -d
```

**Option 2: Kubernetes (Production Scale)**

Use the provided Kubernetes manifests in the deployment guide.

**For comprehensive deployment instructions:**
- [Deployment Guide](./references/deployment.md)

**Production deployment includes:**
- Docker containerization for both API and agent
- GPU allocation for faster synthesis
- Health checks and monitoring
- Horizontal scaling with load bala