fal-audio

What this skill does

## Quick Reference

| STT Model | Endpoint | Speed | Accuracy |
|-----------|----------|-------|----------|
| Whisper | `fal-ai/whisper` | Medium | Highest |
| Whisper Turbo | `fal-ai/whisper-turbo` | Fast | High |
| Whisper Large v3 | `fal-ai/whisper-large-v3` | Slow | Highest |

| TTS Model | Endpoint | Voice Clone | Quality |
|-----------|----------|-------------|---------|
| F5-TTS | `fal-ai/f5-tts` | Yes | High |
| ElevenLabs | `fal-ai/elevenlabs/tts` | Via API | Highest |
| Kokoro | `fal-ai/kokoro/american-english` | No | Good |
| XTTS | `fal-ai/xtts` | Yes | Good |

| Whisper Task | Use Case |
|--------------|----------|
| `transcribe` | Same language text |
| `translate` | Non-English → English |

| Whisper Parameter | Value |
|-------------------|-------|
| `chunk_level` | `"segment"` for timestamps |
| `language` | ISO code (e.g., `"en"`) |

## When to Use This Skill

Use for **audio processing**:
- Transcribing audio/video to text
- Generating subtitles with timestamps
- Translating speech to English
- Cloning voices from reference audio
- Generating natural speech from text

**Related skills:**
- For video with audio: see `fal-text-to-video`
- For API integration: see `fal-api-reference`
- For model comparison: see `fal-model-guide`

---

# fal.ai Audio Models

Complete reference for speech-to-text (STT) and text-to-speech (TTS) models on fal.ai.

## Speech-to-Text Models

### Whisper (OpenAI)
**Endpoint:** `fal-ai/whisper`
**Best For:** Accurate transcription and translation

The industry-standard speech recognition model with support for 99+ languages.

```typescript
import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/speech.mp3",
    task: "transcribe",
    language: "en",
    chunk_level: "segment"
  }
});

console.log(result.text);
console.log(result.chunks);  // With timestamps
```

```python
import fal_client

result = fal_client.subscribe(
    "fal-ai/whisper",
    arguments={
        "audio_url": "https://example.com/speech.mp3",
        "task": "transcribe",
        "language": "en",
        "chunk_level": "segment"
    }
)
print(result["text"])
for chunk in result["chunks"]:
    print(f"[{chunk['timestamp'][0]:.2f}-{chunk['timestamp'][1]:.2f}] {chunk['text']}")
```

**Whisper Parameters:**

| Parameter | Type | Values | Description |
|-----------|------|--------|-------------|
| `audio_url` | string | - | Audio file URL |
| `task` | string | "transcribe", "translate" | Transcribe or translate to English |
| `language` | string | ISO code | Source language (optional, auto-detected) |
| `chunk_level` | string | "segment" | Return timestamps |
| `version` | string | "3" | Whisper version |

**Response Structure:**

```typescript
interface WhisperOutput {
  text: string;  // Full transcription
  chunks?: Array<{
    text: string;
    timestamp: [number, number];  // [start, end] in seconds
  }>;
}
```

### Whisper Turbo
**Endpoint:** `fal-ai/whisper-turbo`
**Best For:** Fast transcription

```typescript
const result = await fal.subscribe("fal-ai/whisper-turbo", {
  input: {
    audio_url: "https://example.com/podcast.mp3",
    task: "transcribe"
  }
});
```

### Whisper Large v3
**Endpoint:** `fal-ai/whisper-large-v3`
**Best For:** Maximum accuracy

```typescript
const result = await fal.subscribe("fal-ai/whisper-large-v3", {
  input: {
    audio_url: "https://example.com/meeting.mp3",
    task: "transcribe",
    language: "en"
  }
});
```

### Whisper Usage Examples

**Transcription with Timestamps:**
```typescript
const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: audioUrl,
    task: "transcribe",
    chunk_level: "segment"
  }
});

// Format as SRT subtitles
result.chunks.forEach((chunk, i) => {
  const start = formatTime(chunk.timestamp[0]);
  const end = formatTime(chunk.timestamp[1]);
  console.log(`${i + 1}\n${start} --> ${end}\n${chunk.text}\n`);
});

function formatTime(seconds: number): string {
  const h = Math.floor(seconds / 3600);
  const m = Math.floor((seconds % 3600) / 60);
  const s = Math.floor(seconds % 60);
  const ms = Math.floor((seconds % 1) * 1000);
  return `${h.toString().padStart(2, '0')}:${m.toString().padStart(2, '0')}:${s.toString().padStart(2, '0')},${ms.toString().padStart(3, '0')}`;
}
```

**Translation (Non-English to English):**
```typescript
const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/french-speech.mp3",
    task: "translate",  // Translates to English
    language: "fr"
  }
});

console.log(result.text);  // English translation
```

**Multi-Language Detection:**
```typescript
// Whisper auto-detects language if not specified
const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/unknown-language.mp3",
    task: "transcribe"
    // language omitted - auto-detect
  }
});
```

## Text-to-Speech Models

### F5-TTS
**Endpoint:** `fal-ai/f5-tts`
**Best For:** Voice cloning from reference audio

```typescript
const result = await fal.subscribe("fal-ai/f5-tts", {
  input: {
    gen_text: "Hello! Welcome to our product demonstration. We're excited to show you what we've built.",
    ref_audio_url: "https://example.com/voice-sample.wav",
    ref_text: "This is a sample of my voice for cloning purposes.",
    model_type: "F5-TTS"
  }
});

console.log(result.audio_url);
```

```python
result = fal_client.subscribe(
    "fal-ai/f5-tts",
    arguments={
        "gen_text": "Hello! Welcome to our product.",
        "ref_audio_url": "https://example.com/voice-sample.wav",
        "ref_text": "This is a sample of my voice."
    }
)
print(result["audio_url"])
```

**F5-TTS Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `gen_text` | string | Text to synthesize |
| `ref_audio_url` | string | Reference voice audio URL |
| `ref_text` | string | Transcript of reference audio |
| `model_type` | string | "F5-TTS" or "E2-TTS" |
| `remove_silence` | boolean | Remove silence from output |

### ElevenLabs TTS
**Endpoint:** `fal-ai/elevenlabs/tts`
**Best For:** Premium voice quality

```typescript
const result = await fal.subscribe("fal-ai/elevenlabs/tts", {
  input: {
    text: "Welcome to fal.ai! Let me tell you about our amazing AI models.",
    voice_id: "21m00Tcm4TlvDq8ikWAM",  // ElevenLabs voice ID
    model_id: "eleven_multilingual_v2"
  }
});

console.log(result.audio.url);
```

**ElevenLabs Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `text` | string | Text to synthesize |
| `voice_id` | string | ElevenLabs voice ID |
| `model_id` | string | TTS model version |
| `stability` | number | Voice stability (0-1) |
| `similarity_boost` | number | Voice similarity (0-1) |

**ElevenLabs Voice IDs (examples):**
- `21m00Tcm4TlvDq8ikWAM` - Rachel (female)
- `AZnzlk1XvdvUeBnXmlld` - Domi (female)
- `EXAVITQu4vr4xnSDxMaL` - Bella (female)
- `ErXwobaYiN019PkySvjV` - Antoni (male)
- `VR6AewLTigWG4xSOukaG` - Arnold (male)

### Kokoro TTS
**Endpoint:** `fal-ai/kokoro/american-english`
**Best For:** Multi-language, natural sounding

```typescript
const result = await fal.subscribe("fal-ai/kokoro/american-english", {
  input: {
    text: "This is a test of the Kokoro text-to-speech system.",
    voice: "af_bella"  // Voice style
  }
});

console.log(result.audio.url);
```

**Kokoro Variants:**
- `fal-ai/kokoro/american-english` - American English
- `fal-ai/kokoro/british-english` - British English
- `fal-ai/kokoro/japanese` - Japanese
- `fal-ai/kokoro/mandarin` - Mandarin Chinese

**Kokoro Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `text` | string | Text to synthesize |
| `voice` | string | Voice style identifier |
| `speed` | number | Speech speed multiplier |

### XTTS (Coqui)
**Endpoint:** `fal-ai/xtts`
**Best For:** Open-source voice cloning

```typescript
const result = await fal.s