twilio-agent-connect

What this skill does

# Twilio Agent Connect

## Overview

Twilio Agent Connect (TAC) is a Python and TypeScript SDK that integrates third-party LLM agentic applications with Twilio's communication technologies. TAC provides middleware for identity resolution, memory/context management (via Conversation Memory), conversation orchestration (via Conversation Orchestrator), and multi-channel handling (Voice, SMS, RCS, WhatsApp, Chat).

**Key Architecture Principle**: TAC is not an agent runtime itself—it's middleware that enables existing LLM applications (OpenAI Agents SDK, Bedrock, LangChain, Microsoft Foundry, etc.) to leverage Twilio Conversations services.

## Product Context

### Core Twilio Conversations Services

TAC integrates with three core Twilio Conversations services:

1. **Conversation Memory (Memory Store)** - Persistent user context and memory management
   - Profile storage with traits and attributes
   - Observation and summary storage
   - Session history with full conversation context
   - Identity resolution (profile lookup by phone/email)

2. **Conversation Orchestrator** - Multi-channel conversation lifecycle management
   - Unified conversation API across all channels
   - Participant management
   - Communication routing
   - Conversation grouping and configuration

3. **Enterprise Knowledge** - Knowledge base integration
   - Semantic search across knowledge bases
   - RAG (Retrieval-Augmented Generation) support
   - Knowledge chunk retrieval with relevance scoring

### Supported Channels

TAC provides built-in support for:
- **Voice** - ConversationRelay (WebSocket-based real-time voice)
- **SMS** - Text messaging
- **RCS** - Rich Communication Services
- **WhatsApp** - WhatsApp Business messaging
- **Chat** - Web chat integrations

All channels support both inbound (customer-initiated) and outbound (agent-initiated) conversations.

### ConversationRelay-Only Mode

TAC supports a simplified "ConversationRelay-only" mode for getting started with voice conversations without requiring Conversation Orchestrator or Conversation Memory setup. This mode provides:
- TwiML generation
- WebSocket protocol handling
- Voice conversation lifecycle management
- Callback-based message processing

## Installation

### Python SDK

**Requirements**: Python 3.10+

```bash
# Using uv (recommended)
uv add git+https://github.com/twilio/twilio-agent-connect-python.git

# With server support (includes FastAPI and uvicorn for TACFastAPIServer)
uv add git+https://github.com/twilio/twilio-agent-connect-python.git --extra server

# Using pip
pip install git+https://github.com/twilio/twilio-agent-connect-python.git
pip install "git+https://github.com/twilio/twilio-agent-connect-python.git[server]"
```

### TypeScript SDK

**Requirements**: Node.js 22.13+

```bash
# Clone and build (not yet published to npm)
git clone https://github.com/twilio/twilio-agent-connect-typescript.git
cd twilio-agent-connect-typescript
npm install
npm run build
```

## Quick Start

### Multi-Channel Agent with OpenAI (Python)

```python
from dotenv import load_dotenv
from openai import AsyncOpenAI
from tac import TAC, TACConfig
from tac.adapters.openai import with_tac_memory
from tac.channels.sms import SMSChannel
from tac.channels.voice import VoiceChannel
from tac.server import TACFastAPIServer

load_dotenv()

tac = TAC(config=TACConfig.from_env())
voice_channel = VoiceChannel(tac)
sms_channel = SMSChannel(tac)
openai_client = AsyncOpenAI()

conversation_history = {}
SYSTEM_INSTRUCTIONS = (
    "You are a customer service agent speaking with a user over voice or SMS. "
    "Keep responses short and conversational — a sentence or two. "
    "Do not use markdown, asterisks, bullets, or emojis; your words will be "
    "spoken aloud or sent as plain text."
)

async def handle_message_ready(user_message, context, memory_response):
    conv_id = context.conversation_id

    if conv_id not in conversation_history:
        conversation_history[conv_id] = []
    conversation_history[conv_id].append({"role": "user", "content": user_message})

    # Inject conversation memory and profile into OpenAI client
    client = with_tac_memory(openai_client, memory_response, context)

    response = await client.responses.create(
        model="gpt-5.4-mini",
        instructions=SYSTEM_INSTRUCTIONS,
        input=conversation_history[conv_id]
    )

    llm_response = response.output_text
    conversation_history[conv_id].append({"role": "assistant", "content": llm_response})

    return llm_response

tac.on_message_ready(handle_message_ready)
TACFastAPIServer(tac=tac, voice_channel=voice_channel, messaging_channels=[sms_channel]).start()
```

### Multi-Channel Agent with OpenAI (TypeScript)

```typescript
import { config } from 'dotenv';
import OpenAI from 'openai';
import {
  TAC,
  TACConfig,
  VoiceChannel,
  SMSChannel,
  TACServer,
  MemoryPromptBuilder,
} from 'twilio-agent-connect';

config();

const openai = new OpenAI();
const tac = await TAC.create({ config: TACConfig.fromEnv() });
const voiceChannel = new VoiceChannel(tac);
const smsChannel = new SMSChannel(tac);

tac.registerChannel(voiceChannel);
tac.registerChannel(smsChannel);

const conversationHistory: Record<string, OpenAI.Chat.ChatCompletionMessageParam[]> = {};

const SYSTEM_INSTRUCTIONS =
  'You are a customer service agent speaking with a user over voice or SMS. ' +
  'Keep responses short and conversational — a sentence or two. ' +
  'Do not use markdown, asterisks, bullets, or emojis; your words will be ' +
  'spoken aloud or sent as plain text.';

tac.onMessageReady(async ({ conversationId, message, memory, session }) => {
  const convId = conversationId as string;

  if (!conversationHistory[convId]) {
    conversationHistory[convId] = [];
  }

  const memoryContext = MemoryPromptBuilder.build(memory, session);
  const systemPrompt = SYSTEM_INSTRUCTIONS + (memoryContext ? `\n\n${memoryContext}` : '');

  conversationHistory[convId].push({ role: 'user', content: message });

  const response = await openai.chat.completions.create({
    model: 'gpt-5.4-mini',
    messages: [
      { role: 'system', content: systemPrompt },
      ...conversationHistory[convId],
    ],
  });

  const llmResponse = response.choices[0]?.message?.content ?? '';
  conversationHistory[convId].push({ role: 'assistant', content: llmResponse });

  return llmResponse;
});

const server = new TACServer(tac);
await server.start();
```

## Configuration

### Required Environment Variables

```bash
# Twilio Account Credentials
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_API_KEY=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_SECRET=your_api_key_secret

# Conversation Configuration
TWILIO_CONVERSATION_CONFIGURATION_ID=conv_configuration_xxxx

# Phone Number
TWILIO_PHONE_NUMBER=+1234567890

# Server Configuration (for Voice)
TWILIO_VOICE_PUBLIC_DOMAIN=your-domain.ngrok.io
```

### Optional Memory Configuration

```bash
# Conversation Memory (optional)
TWILIO_MEMORY_STORE_ID=mem_service_xxxx
TWILIO_TRAIT_GROUPS=Contact,Preferences
```

## Cloud Platform Integrations

### AWS Integration

**Package**: `twilio-agent-connect-aws`

Connect AWS agent services to Twilio channels:

```bash
# With Strands SDK
pip install twilio-agent-connect-aws[strands,server]

# With Bedrock Agents
pip install twilio-agent-connect-aws[bedrock,server]

# With Bedrock AgentCore
pip install twilio-agent-connect-aws[agentcore,server]
```

**Features**:
- **StrandsConnector** - AWS Strands SDK integration with per-conversation agent isolation
- **BedrockConnector** - AWS Bedrock Agents (console-created agents)
- **BedrockAgentCoreConnector** - AWS Bedrock AgentCore (custom agent code deployment)

**Repository**: https://github.com/twilio/twilio-agent-connect-aws

### Microsoft/Azure Integration

**Package**: `twilio-agent-connect-microsoft` (formerly `tac-azure`)

Connect Microsoft Foundry agents to Twilio channels:

```bash
# With Agent Framework
p