audio-generation

Included with Lifetime

$97 forever

Guide to audio generation and understanding in MassGen. Covers text-to-speech, music, sound effects, and audio understanding across ElevenLabs and OpenAI backends.

Image & Video

What this skill does


# Audio Generation

Generate audio using `generate_media` with `mode="audio"`. Supports speech (TTS), music, and sound effects. ElevenLabs is preferred when available, with OpenAI as fallback.

## Quick Start

```python
# Text-to-speech (auto-selects ElevenLabs if key available)
generate_media(prompt="Hello, welcome to our presentation!", mode="audio")

# With specific voice
generate_media(prompt="Hello!", mode="audio", voice="Rachel")

# Music generation (ElevenLabs only)
generate_media(prompt="Upbeat jazz piano with soft drums", mode="audio",
               audio_type="music", duration=30)

# Sound effects (ElevenLabs only)
generate_media(prompt="Thunder rolling across a mountain valley", mode="audio",
               audio_type="sound_effect", duration=5)
```

## Audio Types

| Type | Backends | Description |
|------|----------|-------------|
| `"speech"` (default) | ElevenLabs, OpenAI | Text-to-speech with voice selection |
| `"music"` | ElevenLabs only | Music generation from text prompt |
| `"sound_effect"` | ElevenLabs only | Sound effect generation |
| `"voice_conversion"` | ElevenLabs only | Change voice of existing audio (speech-to-speech) |
| `"audio_isolation"` | ElevenLabs only | Remove background noise, isolate vocals |
| `"voice_design"` | ElevenLabs only | Create a new synthetic voice from text description |
| `"voice_clone"` | ElevenLabs only | Clone a voice from audio samples |
| `"dubbing"` | ElevenLabs only | Translate and dub audio to another language |

## Backend Comparison

| Backend | Default Model | Supports | API Key |
|---------|--------------|----------|---------|
| **ElevenLabs** (priority 1) | `eleven_multilingual_v2` | Speech, music, SFX | `ELEVENLABS_API_KEY` |
| **OpenAI** (priority 2) | `gpt-4o-mini-tts` | Speech only | `OPENAI_API_KEY` |

If ElevenLabs TTS fails, the system automatically falls back to OpenAI TTS.

## Key Parameters

| Parameter | Description | Example |
|-----------|-------------|---------|
| `prompt` | Text to speak (speech) or description (music/SFX) | `"Hello world!"` |
| `voice` | Voice name or ID | `"Rachel"`, `"nova"`, `"alloy"` |
| `audio_type` | Type of audio | `"speech"`, `"music"`, `"sound_effect"` |
| `duration` | Length in seconds (music/SFX only) | `30` |
| `instructions` | Speaking style (OpenAI `gpt-4o-mini-tts` only) | `"warm, reflective tone"` |
| `audio_format` | Output format | `"mp3"`, `"wav"`, `"opus"` |

## Voice Quick Reference

**ElevenLabs** (top voices):
| Voice | Character |
|-------|-----------|
| Rachel | Warm, conversational female |
| Sarah | Clear, professional female |
| Josh | Friendly male |
| Adam | Deep, authoritative male |
| Emily | Bright, energetic female |

**OpenAI** voices: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer`, `coral`, `sage`

## Important: prompt vs instructions

For speech, `prompt` is the **literal text to speak**. Style guidance goes in `instructions`:

```python
# CORRECT: prompt = text to speak, instructions = how to speak it
generate_media(
    prompt="Welcome to the annual report presentation.",
    mode="audio",
    voice="alloy",
    instructions="warm, reflective tone with measured pacing",
    backend_type="openai"
)

# WRONG: Don't put style instructions in prompt
generate_media(prompt="Say this warmly: Welcome...", mode="audio")  # Bad!
```

`instructions` only works with OpenAI `gpt-4o-mini-tts`. ElevenLabs uses voice selection for tone.

## Audio Understanding

Use `read_media` (not `generate_media`) to analyze existing audio:

```python
read_media(path="recording.mp3", prompt="Transcribe and summarize this audio")
```

## Need More Control?

- **Full ElevenLabs voice catalog (28+ voices)**: See [references/voices.md](references/voices.md)
- **Music and sound effects details**: See [references/music_and_sfx.md](references/music_and_sfx.md)
- **Advanced audio capabilities (voice conversion, cloning, isolation, dubbing)**: See [references/advanced.md](references/advanced.md)

Files: 4

Size: 16.0 KB

Complexity: 31/100

Category: Image & Video

Source: https://github.com/massgen/massgen/tree/main/massgen/skills/audio-generation

Related in Image & Video

watch

Included

Watch a video (URL or local path). Downloads with yt-dlp, extracts auto-scaled frames with ffmpeg, pulls the transcript from captions (or Whisper API fallback), and hands the result to Claude so it can answer questions about what's in the video.

Image & Videoscriptsfeatured

physical-ai-defect-image-generation

Included

Use when the user wants to orchestrate defect image generation, run associated setup, or handle outputs on OSMO. The Day 0 path handles cold-start with USD-to-ROI, image-edit augmentation, and AnomalyGen to create initial PCBA datasets. The Day 1 path performs inference and labeling on real images. This skill helps with first-time asset setup, creation of finetuning checkpoints, and configuring deployment. Trigger keywords: defect image generation, dig workflow, dig pipeline, defect image detection workflow, aoi pipeline, aoi anomalygen, usd2roi anomalygen, day 0 pcba, day 1 pcba, day 1 real-photo alignment, day 1 manual roi, metal surface anomaly, glass defect, anomalygen finetune, setup_pcb, setup_metal, setup_glass, setup_pretrained, dig setup, dig datasets, dig pretrained checkpoint, dig image-edit endpoint.

Image & Videoscripts

accelint-react-best-practices

Included

React performance optimization and best practices. ALWAYS use this skill when working with any React code - writing components, hooks, JSX; refactoring; optimizing re-renders, memoization, state management; reviewing for performance; fixing hydration mismatches; debugging infinite re-renders, stale closures, input focus loss, animations restarting; preventing remounting; implementing transitions, lazy initialization, effect dependencies. Even simple React tasks benefit from these patterns. Covers React 19+ (useEffectEvent, Activity, ref props). Triggers - useEffect, useState, useMemo, useCallback, memo, inline components, nested components, components inside components, re-render, performance, hydration, SSR, Next.js, useDeferredValue, combined hooks.

Image & Videoscripts

elevenlabs-agents

Included

Build conversational AI voice agents with ElevenLabs Platform using React, JavaScript, React Native, or Swift SDKs. Configure agents, tools (client/server/MCP), RAG knowledge bases, multi-voice, and Scribe real-time STT. Use when: building voice chat interfaces, implementing AI phone agents with Twilio, configuring agent workflows or tools, adding RAG knowledge bases, testing with CLI "agents as code", or troubleshooting deprecated @11labs packages, Android audio cutoff, CSP violations, dynamic variables, or WebRTC config. Keywords: ElevenLabs Agents, ElevenLabs voice agents, AI voice agents, conversational AI, @elevenlabs/react, @elevenlabs/client, @elevenlabs/react-native, @elevenlabs/elevenlabs-js, @elevenlabs/agents-cli, elevenlabs SDK, voice AI, TTS, text-to-speech, ASR, speech recognition, turn-taking model, WebRTC voice, WebSocket voice, ElevenLabs conversation, agent system prompt, agent tools, agent knowledge base, RAG voice agents, multi-voice agents, pronunciation dictionary, voice speed control, elevenlabs scribe, @11labs deprecated, Android audio cutoff, CSP violation elevenlabs, dynamic variables elevenlabs, case-sensitive tool names, webhook authentication

Image & Videoscripts

humanizer

Included

Humanize AI-generated text by detecting and removing patterns typical of LLM output. Rewrites text to sound natural, specific, and human. Uses 28 pattern detectors, 560+ AI vocabulary terms across 3 tiers, and statistical analysis (burstiness, type-token ratio, readability) for comprehensive detection. Use when asked to humanize text, de-AI writing, make content sound more natural/human, review writing for AI patterns, score text for AI detection, or improve AI-generated drafts. Covers content, language, style, communication, and filler categories.

Image & Videoscripts

generating-mermaid-diagrams

Included

Salesforce architecture diagrams using Mermaid with ASCII fallback. Use this skill when generating text-based diagrams for Salesforce architecture, OAuth flows, ERDs, integration sequences, or Agentforce structure. TRIGGER when: user says "diagram", "visualize", "ERD", or asks for sequence diagrams, flowcharts, class diagrams, or architecture visualizations in Mermaid. DO NOT TRIGGER when: user wants PNG/SVG image output (use generating-visual-diagrams), or asks about non-Salesforce systems.

Image & Videoscripts