vss-summarize-video

What this skill does

## Instructions

Follow the routing tables and step-by-step workflows below. Each section that ends in *workflow*, *quick start*, or *flow* is intended to be executed top-to-bottom. Detailed reference material lives in `references/` and helper scripts live in `scripts/` — call them via `run_script` when the skill points to a script by name.

## Examples

Worked end-to-end examples are kept under `evals/` (each `*.json` manifest contains a runnable scenario) and inline in the per-workflow `curl` blocks below. Run a Tier-3 evaluation with `nv-base validate <this-skill-dir> --agent-eval` to replay them.

You are a video summarization assistant. You call the VLM NIM or the video summarization
microservice **directly**. Always run `curl` commands yourself; never instruct the user to run them.

Primary video workflow query type: **"Summarize this video."** Direct video summarization API
and service-ops requests are handled by the reference-routed sections below.

## Purpose

Produce a single, polished narrative summary of one recorded video clip, with
timestamped events when the LVS microservice path is reachable.

**Do NOT use this skill for:**
- Live RTSP captioning — use `vss-deploy-dense-captioning`.
- Incident-range or alert-window reports — use `vss-generate-video-report` Mode B.
- Semantic search across the archive — use `vss-search-archive`.

## Prerequisites

- VSS `lvs` profile running on `$HOST_IP` (port 38111) OR a reachable
  VLM/RT-VLM endpoint as a fallback. The `vss-deploy-profile` skill brings
  these up.
- Network reachability from the agent host to both endpoints; clip URLs from
  VIOS must be fetchable by the chosen backend.
- `jq` and `curl` available on the agent host.

## Limitations

- Direct VLM fallback uses a single fixed prompt and cannot target
  scenario/events — output quality is lower than the LVS path.
- Remote VLM endpoints generally cannot reach `localhost`/private clip URLs.
- One backend call per request; no parallel hedging or multi-pass summaries.

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `/v1/ready` returns 503 repeatedly | LVS service still warming up | Retry up to ~30 s as shown in *Setup*; if it never returns 200 the service may not be deployed |
| Empty `video_summary` and `events` | Clip does not contain the requested events | Re-run with broader `scenario` or different `events` |
| VLM returns `<think>` block | Cosmos Reason 2 reasoning mode | Strip everything up to `</think>` before rendering |
| Empty stdout from `curl /v1/ready` | Service legitimately returns 200 with empty body | Always check HTTP status with `-o /dev/null -w '%{http_code}'`, never inspect the body |

See [`references/video-summarization-debugging.md`](references/video-summarization-debugging.md) for deeper diagnostics.

## Reference Map

Use these references only when the user asks for the relevant detail, or when
the core workflow below needs deeper video summarization information:

- **video summarization API details**: [`references/video-summarization-api.md`](references/video-summarization-api.md) for
  `/v1/summarize`, `/summarize`, `/v1/generate_captions`,
  `/v1/stream_summarize`, health probes, `/models`, `/recommended_config`,
  `/metrics`, request fields, response shapes, and API gotchas.
- **video summarization service configuration and ops**:
  [`references/video-summarization-deployment.md`](references/video-summarization-deployment.md) for
  the VSS `lvs` profile, ports, required env vars, logs, status, dry-runs,
  teardown, model/backend swaps, Elasticsearch/Neo4j/ArangoDB backend
  selection, and service-level troubleshooting.
- **Extended video summarization ops references**:
  [`references/video-summarization-environment-variables.md`](references/video-summarization-environment-variables.md),
  [`references/video-summarization-debugging.md`](references/video-summarization-debugging.md), and
  `assets/video-summarization.env.example`.

Load `video-summarization-api.md` only when you need a request field, response shape, or
endpoint that is not already covered by the Step 2 LVS or fallback VLM
example below, or when handling a direct video summarization API
request. Load `video-summarization-deployment.md` only for deployment,
configuration, or service operations.

## Video Summarization API And Service Ops Requests

If the user asks to call or debug video summarization endpoints directly, answer from
[`references/video-summarization-api.md`](references/video-summarization-api.md) instead of running the
end-to-end video summarization workflow. Examples: list video summarization models, check
readiness, get recommended chunking config, inspect metrics, explain a 422
response, or build a `/v1/summarize` request body.

If the user asks to configure, deploy, restart, tear down, or troubleshoot the
video summarization service, prefer the `vss-deploy-profile` skill for full VSS profile
deployment and use [`references/video-summarization-deployment.md`](references/video-summarization-deployment.md)
for video summarization-specific service details.

## Routing

Decide purely from video summarization service availability (probed in
*Setup → Availability checks* below). **Duration does not drive routing.**

| `/v1/ready` | Backend | Endpoint |
|---|---|---|
| HTTP 200 | LVS microservice with HITL | `POST ${LVS_BACKEND_URL}/v1/summarize` |
| Anything else | VLM / RT-VLM with the default prompt + fallback note | `POST ${VLM_BASE_URL}/v1/chat/completions` |

Fallback message when the LVS service is unreachable — copy verbatim above the summary:

> ⚠ **Note:** Input video `<name>` is `<N>`s long.
> The video summarization service is not deployed, so this summary was
> produced by the VLM alone with a generic default prompt. Deploy the
> `lvs` profile for higher-quality summaries with scenario/events
> targeting.

## Deployment prerequisite

The VSS **lvs** profile on `$HOST_IP` is the primary backend. If the
`/v1/ready` probe (see *Setup → Availability checks*) returns anything
other than 200 after the warmup retries, ask the user:

> *"The VSS `lvs` profile isn't running on `$HOST_IP`. Shall I deploy it now using the `/vss-deploy-profile` skill with `-p lvs`? Reply `no` to summarize with the VLM-only fallback instead (lower quality, no scenario/events targeting)."*

- **Yes** → hand off to `/vss-deploy-profile`, then re-probe and continue with Step 2 (LVS + HITL).
- **No** → go straight to **Step 2 fallback (VLM with default prompt)** and prepend the Routing fallback note. Do not ask again, and do not run scenario/events HITL.
- **Pre-authorized to deploy autonomously** (caller said so explicitly) → skip the confirmation and invoke `/vss-deploy-profile` directly.
- **Pre-authorized to use VLM fallback** ("skip lvs, just use the VLM") → go straight to Step 2 fallback without prompting.

---

## Setup

**Endpoints (defaults for a local VSS `lvs` deployment):**

- VLM / RT-VLM: `${VLM_BASE_URL}` — default `${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}`
- LVS service: `${LVS_BACKEND_URL}` — default `http://${HOST_IP:-localhost}:38111`
- VIOS: owned by `vss-manage-video-io-storage`; refer there.

Use env vars when set (strip trailing `/v1` from the VLM base — the skill appends it). Otherwise use the defaults. If neither works, ask the user — do not scan ports or read config files to guess.

**Model name:** read `${VLM_NAME}` (default
`nim_nvidia_cosmos-reason2-8b_hf-1208`). It must match the id RT-VLM
`/v1/models` advertises; do not substitute the friendly
`nvidia/cosmos-reason2-8b`.

For endpoint schemas, optional fields, response envelopes, and error handling, see [`references/video-summarization-api.md`](references/video-summarization-api.md).

**Availability checks** (run both before routing).
**Readiness is determined by the HTTP status code only** — the LVS
`/v1/ready` may legitimately return `200` with an empty body, so do not
inspect the body.

```bash
VLM="${VLM_BASE_URL:-${RTVI_VLM_BASE_URL:-http://