video-understand

What this skill does

# Video Understanding Skill

This skill provides specialized video understanding functionality using the z-ai-web-dev-sdk package, enabling AI models to analyze, describe, and extract information from video content including motion, temporal sequences, and scene changes.

## Skills Path

**Skill Location**: `{project_path}/skills/video-understand`

this skill is located at above path in your project.

**Reference Scripts**: Example test scripts are available in the `{Skill Location}/scripts/` directory for quick testing and reference. See `{Skill Location}/scripts/video-understand.ts` for a working example.

## Overview

Video Understanding focuses specifically on video content analysis, providing capabilities for:
- Video scene understanding and description
- Action and motion detection
- Temporal sequence analysis
- Event detection in videos
- Video content summarization
- Scene change detection
- People and object tracking across frames
- Audio-visual content analysis (when applicable)

**IMPORTANT**: z-ai-web-dev-sdk MUST be used in backend code only. Never use it in client-side code.

## Prerequisites

The z-ai-web-dev-sdk package is already installed. Import it as shown in the examples below.

## CLI Usage (For Simple Tasks)

For quick video analysis tasks, you can use the z-ai CLI instead of writing code. This is ideal for simple video descriptions, testing, or automation.

### Basic Video Analysis

```bash
# Analyze a video from URL
z-ai vision --prompt "Summarize what happens in this video" --image "https://example.com/video.mp4"

# Note: Use --image flag for video URLs as well
z-ai vision -p "Describe the key events" -i "https://example.com/presentation.mp4"
```

### Analyze Local Videos

```bash
# Analyze a local video file
z-ai vision -p "What activities are shown in this video?" -i "./recording.mp4"

# Save response to file
z-ai vision -p "Provide a detailed summary" -i "./meeting.mp4" -o summary.json
```

### Advanced Video Analysis

```bash
# Complex scene understanding with thinking
z-ai vision \
  -p "Analyze this video and identify: 1) Main events, 2) People and their actions, 3) Timeline of key moments" \
  -i "./event.mp4" \
  --thinking \
  -o analysis.json

# Action detection
z-ai vision \
  -p "Identify all actions performed by people in this video" \
  -i "./sports.mp4" \
  --thinking
```

### Streaming Output

```bash
# Stream the video analysis
z-ai vision -p "Describe this video content" -i "./video.mp4" --stream
```

### CLI Parameters

- `--prompt, -p <text>`: **Required** - Question or instruction about the video
- `--image, -i <URL or path>`: Optional - Video URL or local file path (despite the name, it works for videos too)
- `--thinking, -t`: Optional - Enable chain-of-thought reasoning for complex analysis (default: disabled)
- `--output, -o <path>`: Optional - Output file path (JSON format)
- `--stream`: Optional - Stream the response in real-time

### Supported Video Formats

- MP4 (.mp4) - Most widely supported format
- AVI (.avi) - Audio Video Interleave
- MOV (.mov) - QuickTime format
- WebM (.webm) - Web-optimized format
- MKV (.mkv) - Matroska format
- FLV (.flv) - Flash Video format

### When to Use CLI vs SDK

**Use CLI for:**
- Quick video summaries
- One-off video analysis
- Testing video understanding capabilities
- Simple automation scripts
- Generating video descriptions

**Use SDK for:**
- Multi-turn conversations about videos
- Complex video processing pipelines
- Production applications with error handling
- Custom integration with video processing logic
- Batch video processing with custom workflows

## Recommended Approach

For better performance and reliability with local videos, consider:
1. Uploading videos to a CDN and using URLs
2. For shorter videos, convert key frames to images for faster analysis
3. For long videos, consider chunking or sampling at intervals

## Basic Video Understanding Implementation

### Single Video Analysis

```javascript
import ZAI from 'z-ai-web-dev-sdk';

async function analyzeVideo(videoUrl, prompt) {
  const zai = await ZAI.create();

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'text',
            text: prompt
          },
          {
            type: 'video_url',
            video_url: {
              url: videoUrl
            }
          }
        ]
      }
    ],
    thinking: { type: 'disabled' }
  });

  return response.choices[0]?.message?.content;
}

// Usage examples
const summary = await analyzeVideo(
  'https://example.com/presentation.mp4',
  'Summarize the key points presented in this video'
);

const actionDetection = await analyzeVideo(
  'https://example.com/sports.mp4',
  'Identify and describe all athletic actions performed in this video'
);
```

### Video Scene Understanding

```javascript
import ZAI from 'z-ai-web-dev-sdk';

async function understandVideoScenes(videoUrl) {
  const zai = await ZAI.create();

  const prompt = `Analyze this video and provide:
1. Overall summary of the video content
2. Main scenes or segments (with approximate timestamps if possible)
3. Key people or characters and their roles
4. Important actions or events in chronological order
5. Setting and environment description
6. Overall mood or tone`;

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          { type: 'text', text: prompt },
          { type: 'video_url', video_url: { url: videoUrl } }
        ]
      }
    ],
    thinking: { type: 'enabled' } // Enable for detailed analysis
  });

  return response.choices[0]?.message?.content;
}

// Usage
const sceneAnalysis = await understandVideoScenes(
  'https://example.com/documentary.mp4'
);
```

### Motion and Action Detection

```javascript
import ZAI from 'z-ai-web-dev-sdk';

async function detectActions(videoUrl, specificAction = null) {
  const zai = await ZAI.create();

  const prompt = specificAction
    ? `Identify all instances of "${specificAction}" in this video. For each instance, describe when it occurs and provide details about how it's performed.`
    : 'Identify and describe all significant actions and movements in this video. Include who is performing them and when they occur.';

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          { type: 'text', text: prompt },
          { type: 'video_url', video_url: { url: videoUrl } }
        ]
      }
    ],
    thinking: { type: 'enabled' }
  });

  return response.choices[0]?.message?.content;
}

// Usage
const runningActions = await detectActions(
  'https://example.com/sports.mp4',
  'running'
);

const allActions = await detectActions(
  'https://example.com/activity.mp4'
);
```

### Event Timeline Extraction

```javascript
import ZAI from 'z-ai-web-dev-sdk';

async function extractTimeline(videoUrl) {
  const zai = await ZAI.create();

  const prompt = `Create a detailed timeline of events in this video:
- Identify key moments and transitions
- Note approximate timing (beginning, middle, end or specific timestamps if visible)
- Describe what happens at each key point
- Identify any cause-and-effect relationships between events

Format as a chronological list.`;

  const response = await zai.chat.completions.createVision({
    messages: [
      {
        role: 'user',
        content: [
          { type: 'text', text: prompt },
          { type: 'video_url', video_url: { url: videoUrl } }
        ]
      }
    ],
    thinking: { type: 'enabled' }
  });

  return response.choices[0]?.message?.content;
}
```

### Video Content Classification

```javascript
import ZAI from 'z-ai-web-dev-sdk';

async function classifyVideo(videoUrl) {
  const zai = await ZAI.create();

  const prompt = `Classify this video content:
1. Primary category (e.g., educational, entertainment, sports, news, tuto