ai-voice-room

What this skill does

# AI Voice Room

Dispatch an AI voice agent into a LiveKit video room as a real participant. The agent listens to room audio, transcribes speech via Deepgram (from the `transcription` skill), processes it through an LLM (via `ai-core`'s `getModel()`), generates a spoken response via TTS, and publishes audio back to the room. Includes predefined personas (note-taker, strategist, interviewer, tutor) and a management API for creating and removing agents.

## Prerequisites

- Next.js app with `src/` directory and App Router
- `video-room` skill installed (LiveKit room + participant tokens)
- `ai-core` skill installed (`getModel()` at `@/lib/ai`)
- `transcription` skill installed (Deepgram client at `@/lib/video/transcription`)
- `env-config` skill installed (`src/env.ts`)
- shadcn/ui initialized

## Installation

No new packages required. Uses packages already installed by dependencies:

- `ai` (from ai-core)
- `livekit-server-sdk` (from video-room)
- `@deepgram/sdk` (from transcription)

## Environment Variables

No new environment variables required. Uses existing variables from dependencies:

- `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` (from video-room)
- `LIVEKIT_URL` (from video-room)
- `DEEPGRAM_API_KEY` (from transcription)
- `AI_GATEWAY_API_KEY` (from ai-core)

## What Gets Created

```
src/
├── lib/
│   └── ai/
│       ├── voice-agent.ts                # Server: create, dispatch, remove AI agents
│       ├── agent-personas.ts             # Predefined agent personas with system prompts
│       └── types-voice-agent.ts          # VoiceAgentConfig, AgentPersona, AgentState
├── app/
│   └── api/
│       └── ai/
│           └── voice-agent/
│               └── route.ts              # POST create / DELETE remove agent
└── components/
    └── video/
        └── ai-agent-indicator.tsx        # UI indicator for AI agent state
```

## Architecture

The AI voice agent pipeline runs as follows:

```
Room Audio → Deepgram STT → LLM (via ai-core) → TTS → Room Audio
                                ↑
                          Agent Persona
                        (system prompt)
```

**Important**: The real-time audio pipeline (STT -> LLM -> TTS running continuously) is a background worker process. This skill provides:

1. **Dispatch layer** — API to create/remove agents in rooms
2. **Persona system** — Configurable agent behaviors
3. **Agent token management** — LiveKit participant tokens for AI agents
4. **State tracking** — Track agent state (idle/listening/thinking/speaking)
5. **UI indicator** — Visual feedback for agent state in the room

The actual continuous audio processing loop would run as a separate worker (e.g., a LiveKit Agents framework worker or a custom Node.js process). The dispatch API creates the agent's room presence and signals the worker to start processing.

## Setup Steps

### Step 1: Create `src/lib/ai/types-voice-agent.ts`

```typescript
export type AgentState = "idle" | "listening" | "thinking" | "speaking";

export type AgentPersona = {
  /** Unique identifier for the persona */
  id: string;
  /** Display name shown in the room */
  name: string;
  /** System prompt that shapes the agent's behavior */
  systemPrompt: string;
  /** TTS voice identifier (provider-specific, e.g., "aura-asteria-en" for Deepgram) */
  voice: string;
  /** Whether the agent proactively speaks or only responds when addressed */
  speakingStyle: "proactive" | "reactive";
  /** Optional model override (defaults to ai-core's getModel()) */
  modelId?: string;
};

export type VoiceAgentConfig = {
  /** The LiveKit room to join */
  roomName: string;
  /** The persona to use for this agent */
  persona: AgentPersona;
  /** Agent's participant identity in the room */
  participantIdentity: string;
  /** Agent's display name in the room */
  participantName: string;
};

export type VoiceAgentStatus = {
  /** The agent's current state */
  state: AgentState;
  /** Room the agent is in */
  roomName: string;
  /** Persona being used */
  personaId: string;
  /** Participant identity in the room */
  participantIdentity: string;
  /** When the agent was dispatched */
  dispatchedAt: Date;
};

export type DispatchAgentRequest = {
  roomName: string;
  personaId: string;
};

export type DispatchAgentResponse = {
  participantIdentity: string;
  participantName: string;
  roomName: string;
  personaId: string;
  token: string;
};
```

### Step 2: Create `src/lib/ai/agent-personas.ts`

```typescript
import type { AgentPersona } from "./types-voice-agent";

/**
 * Predefined agent personas for AI voice room participants.
 * Each persona defines the agent's behavior, voice, and speaking style.
 */
export const AGENT_PERSONAS: Record<string, AgentPersona> = {
  "note-taker": {
    id: "note-taker",
    name: "Notetaker",
    systemPrompt: `You are an AI note-taking assistant in a live meeting. Your role:
- Listen carefully to the entire conversation
- Rarely speak unless directly asked a question
- When asked, provide a concise summary of key points discussed so far
- Track action items, decisions, and open questions
- If asked "what did we decide about X?", recall the relevant discussion accurately
- Keep your responses brief (1-2 sentences) unless asked for a full summary
- Never interrupt the flow of conversation
- When summarizing, organize by: Key Decisions, Action Items, Open Questions`,
    voice: "aura-asteria-en",
    speakingStyle: "reactive",
  },

  strategist: {
    id: "strategist",
    name: "Strategist",
    systemPrompt: `You are an AI strategic advisor participating in a meeting. Your role:
- Actively engage in the discussion with thoughtful contributions
- Offer alternative perspectives and identify blind spots
- Challenge assumptions constructively with "have you considered..." prompts
- Synthesize multiple viewpoints into actionable frameworks
- Suggest next steps and prioritization when the discussion stalls
- Keep contributions concise (2-3 sentences max)
- Wait for natural pauses before contributing — don't interrupt
- Reference specific points others have made to show active listening`,
    voice: "aura-orion-en",
    speakingStyle: "proactive",
  },

  interviewer: {
    id: "interviewer",
    name: "Interviewer",
    systemPrompt: `You are an AI interviewer conducting a conversational interview. Your role:
- Ask thoughtful follow-up questions that dig deeper into responses
- Use the STAR method (Situation, Task, Action, Result) to structure follow-ups
- Probe for specifics: "Can you give me a concrete example of that?"
- Listen for gaps or vague answers and ask for clarification
- Maintain a warm, encouraging tone
- Keep questions short and focused (one question at a time)
- After 3-4 follow-ups on a topic, transition to a new area
- Summarize what you've heard before moving on to validate understanding`,
    voice: "aura-luna-en",
    speakingStyle: "proactive",
  },

  tutor: {
    id: "tutor",
    name: "Tutor",
    systemPrompt: `You are an AI tutor participating in a learning session. Your role:
- Explain complex concepts in simple, clear language
- Use analogies and real-world examples to make ideas concrete
- Ask comprehension-check questions: "Does that make sense?" or "Can you explain it back to me?"
- Break down problems into smaller steps
- When someone is confused, try a different explanation approach
- Encourage questions and make it safe to say "I don't understand"
- Build on what the learner already knows
- Keep explanations under 30 seconds of speech — pause for questions`,
    voice: "aura-athena-en",
    speakingStyle: "reactive",
  },
} as const;

/**
 * Get an agent persona by ID.
 * @throws Error if the persona ID is not found.
 */
export function getPersona(personaId: string): AgentPersona {
  const persona = AGENT_PERSONAS[personaId];
  if (!persona) {
    const available = Object.keys(AGENT_PERSONAS).join(", ");
    throw new Error(
      `Unknown persona "${personaId}". Available personas: ${available}`
    );
  }