comfyui-video-production

What this skill does

# ComfyUI Video Production Pipeline

End-to-end video production orchestration for ComfyUI with automatic error recovery, quality validation, and instance management.

## Quick Start: Which Pipeline?

**Creating a multi-shot narrative video?**
→ **Keyframe Pipeline** - Generate keyframes → Animate → Stitch with transitions

**Animating existing images?**
→ **I2V Batch Pipeline** - Load images → Queue I2V jobs → Auto-validate → Combine

**Need smooth transitions between scenes?**
→ **Transition Pipeline** - Crossfades, motion blur, zoom effects via FFmpeg

**ComfyUI stuck or crashed?**
→ **Instance Manager** - Auto-restart, health checks, queue monitoring

**Debugging video issues?**
→ **Validation Suite** - Check resolution, FPS, codec, face consistency, color grading

---

## Core Pipelines

### Pipeline 1: Keyframe-to-Video (Complete Narrative)

**Use when:** Creating story-driven videos with multiple distinct shots

```
1. Keyframe Generation Phase
   - Generate consistent keyframes with IP-Adapter/LoRA
   - Validate face consistency, lighting, pose progression
   - Save to organized directory structure
   - Auto-retry failed generations

2. I2V Animation Phase
   - Queue each keyframe to I2V model (Wan 2.2, LTX-2, AnimateDiff)
   - Monitor progress via ComfyUI API
   - Validate each clip (resolution, fps, duration)
   - Auto-retry with different seeds if failed

3. Concatenation Phase
   - Pre-flight validation (ensure all clips match)
   - Apply transition effects (crossfade, motion blur)
   - FFmpeg encoding with proper codec
   - Export final video with metadata

4. Quality Assurance
   - Face consistency check across clips
   - Color grading consistency
   - Audio sync validation (if applicable)
   - Generate QA report
```

**Expected output:** Single cohesive video with smooth transitions

---

### Pipeline 2: Batch I2V Processing

**Use when:** You have multiple images to animate independently

```
1. Image Discovery
   - Scan directory for source images
   - Validate image specs (resolution, format)
   - Generate processing manifest

2. Parallel I2V Queue
   - Queue all images to ComfyUI with appropriate prompts
   - Stagger submissions to avoid overload
   - Monitor queue depth and ETA

3. Progressive Validation
   - Check each completed video immediately
   - Flag issues (wrong resolution, fps, corruption)
   - Auto-retry flagged videos

4. Export & Organize
   - Move validated videos to output directory
   - Generate index with metadata
   - Create contact sheet (thumbnail preview grid)
```

**Expected output:** Directory of validated animated clips

---

### Pipeline 3: Video Concatenation with Transitions

**Use when:** Combining existing video clips with professional transitions

```
1. Clip Validation
   - Verify all clips exist and are readable
   - Check resolution, fps, codec consistency
   - Report mismatches with fix suggestions

2. Transition Planning
   - Detect scene changes (cut detection)
   - Recommend transition types (crossfade, zoom, pan)
   - Calculate transition timing

3. FFmpeg Pipeline
   - Apply transitions between clips
   - Re-encode with consistent settings
   - Preserve quality (high bitrate, proper codec)

4. Audio Handling
   - Extract audio from clips (if present)
   - Crossfade audio at transitions
   - Sync to final video timeline
```

**Expected output:** Polished video with seamless transitions

---

## Model Support (2026)

### Image-to-Video Models

| Model | Quality | Speed | VRAM | Best For | Notes |
|-------|---------|-------|------|----------|-------|
| **LTX-2** | ★★★★★ | Medium | 16GB+ | **Production 4K video** | Native 4K, audio+video |
| **Wan 2.2 MoE** | ★★★★★ | Slow | 24GB+ | **Film-quality aesthetics** | First+last frame control |
| Wan 2.1 14B | ★★★★ | Slow | 24GB | High quality | Proven, stable |
| Wan 2.1 1.3B | ★★★ | Fast | 8GB | **Quick iteration** | Consumer-friendly |
| AnimateDiff V3 | ★★★ | Fast | 8GB | Infinite length | Motion LoRAs |
| SVD (Stable Video Diffusion) | ★★★ | Medium | 12GB | Short clips | 14-25 frames |

### Transition Effects

| Effect | Use Case | Encoding Cost |
|--------|----------|---------------|
| **Crossfade** | General purpose | Low |
| **Motion blur** | High-motion scenes | Medium |
| **Zoom in/out** | Dramatic emphasis | Medium |
| **Pan left/right** | Scene establishment | Medium |
| **Fade to/from black** | Chapter breaks | Low |
| **Custom LUT** | Color grading | Low |

---

## ComfyUI Instance Management

### Health Monitoring

```python
# Auto-detected issues:
- Queue stalled (no progress for 5+ minutes)
- Memory leak (VRAM usage climbing)
- Process crashed (connection refused)
- API unresponsive (timeout on /queue endpoint)
- Disk full (output directory at capacity)
```

### Auto-Recovery Actions

```python
1. Soft Recovery (no restart)
   - Clear stuck queue items
   - Force garbage collection
   - Unload models from VRAM

2. Hard Recovery (restart required)
   - Save current queue state
   - Kill ComfyUI process gracefully
   - Wait for port release
   - Restart with same config
   - Restore queue from saved state

3. Emergency Fallback
   - Switch to backup ComfyUI instance
   - Redirect queue to instance on different port
   - Continue processing without data loss
```

### Multi-Instance Support

```bash
# Run multiple ComfyUI instances for parallel processing
Instance 1: localhost:8188 (primary - I2V generation)
Instance 2: localhost:8189 (secondary - upscaling/post-processing)
Instance 3: localhost:8190 (backup - standby for failover)

# Load balancing strategy:
- Round-robin for equal workloads
- Priority-based for mixed tasks
- Failover for crashed instances
```

---

## Validation Suite

### Pre-Generation Validation

```python
✓ Check ComfyUI is running and responsive
✓ Verify models are loaded (UNET, VAE, CLIP)
✓ Confirm output directory has sufficient space
✓ Validate source images exist and are readable
✓ Check prompts are non-empty and formatted correctly
✓ Verify workflow JSON is valid
```

### Post-Generation Validation

```python
✓ Video file exists and is non-zero size
✓ Resolution matches expected (e.g., 768x1024)
✓ FPS matches expected (e.g., 16 or 25)
✓ Duration matches expected (e.g., 3-5 seconds)
✓ Codec is compatible (h264, h265)
✓ No corruption (can read all frames)
✓ Face consistency score >0.85 (if character video)
✓ Color histogram within expected range
```

### Quality Metrics

```python
Metrics tracked:
- Face embedding distance (identity consistency)
- Optical flow magnitude (motion smoothness)
- Frame PSNR/SSIM (interpolation quality)
- Color histogram deviation (lighting consistency)
- Audio sync offset (if audio present)
```

---

## Error Handling & Recovery

### Retry Strategies

```python
1. Seed Randomization Retry
   - Failed generation? Try different seed
   - Max 3 attempts per keyframe
   - Track seeds that fail (avoid reuse)

2. Parameter Adjustment Retry
   - CFG too high causing artifacts? Lower it
   - Steps too low causing incompleteness? Increase
   - Resolution too high OOM? Downscale

3. Model Fallback Retry
   - Wan 2.2 14B OOM? Fall back to 1.3B
   - LTX-2 unavailable? Fall back to Wan 2.1
   - AnimateDiff motion broken? Switch motion LoRA

4. Checkpoint Resume
   - Save progress after each successful clip
   - Resume from last successful checkpoint
   - Skip already-generated clips
```

### Failure Logging

```python
logs/
├── 2026-02-16_pipeline.log       # Main pipeline log
├── 2026-02-16_comfyui.log        # ComfyUI stdout/stderr
├── 2026-02-16_validation.json    # Validation results
├── 2026-02-16_failures.json      # Failed attempts with reasons
└── 2026-02-16_recovery.json      # Recovery actions taken
```

---

## Directory Structure

### Organized Output

```
project_name/
├── 00_keyframes/                 # Source keyframe images
│   ├── kf01_scene_description.png
│   ├── kf02_scene_description.png
│   └── ...
├── 01_clips/                     # Individual animated clips
│   ├── clip_0