physical-ai-video-data-augmentation

What this skill does

# Physical AI Video Data Augmentation Workflow Orchestrator

Default workflow skill for VDA execution on OSMO. It owns flow selection,
preflight, cache readiness, inference-path decisions, submit-time interpolation,
monitoring, and output retrieval. Component skills are consult-only.

## Purpose

Run the end-to-end VDA workflow safely and reproducibly from preflight to output
download.

Do NOT use this skill for container-internal tuning-only questions.

## Prerequisites

Confirm these before running preflight or any submit. Missing required secrets
surface as `USER_INPUT_REQUIRED:` from `scripts/preflight_credentials.sh`.

| Requirement | How it is satisfied | Used for |
|---|---|---|
| NGC API key (optional) | `NGC_API_KEY`, `NGC_CLI_API_KEY`, or compatible `nvapi-*` token in `NVIDIA_API_KEY`/`OPENAI_API_KEY`/`VLM_API_KEY`/`LLM_API_KEY` | Optional for `nvcr_io` credential refresh and NGC REST scope probe; default VDA image refs are validated via workflow registry probes |
| Hugging Face token | `HF_TOKEN` (or `HUGGING_FACE_HUB_TOKEN`), or a cached token at `~/.cache/huggingface/token` | Creates the OSMO `hf_token` credential; pulls gated Cosmos/SeedVR weights |
| OSMO CLI access | `osmo` on `PATH`, logged in, with a default profile and a registered DATA credential profile matching `storage_url` | Submitting/monitoring workflows and listing/downloading objects |
| GPU pool | At least one `ONLINE` pool in `osmo pool list --mode free`; `POD_TEMPLATE` carries GPU toleration/selectors | Scheduling setup + worker tasks |

Optional (only for the strict NGC org/team probe): `NGC_ORG` + `NGC_TEAM`
(or `NGC_CLI_ORG` / `NGC_CLI_TEAM`). External VLM/LLM endpoint keys are validated
separately, not by preflight.

Key handling rule: `nvapi-*` tokens are first-class inputs for `nvcr_io`.
Never reject by token prefix alone; use workflow registry probe results as
source of truth.

## Instructions

1. Select the workflow (`auto_labeling`, `augmentation_and_al`, `e2e`,
   `e2e_super_resolution`) from user intent.
2. Provide a tentative execution-time overview before starting run actions.
3. Run preflight and readiness checks before submit.
4. Derive submit-time values from the active dataset backend (never guess
   `storage_url`).
5. Submit the workflow with explicit interpolation values and monitor to completion.
6. Retrieve outputs, provide side-by-side comparison evidence for augmented
   flows, and summarize task outcomes.

Use `run_script(...)` for script execution. Canonical examples:

```python
run_script("bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/augmentation_and_al.yaml")
run_script("python3 scripts/pre_submit_guard.py --workflow assets/configs/osmo/auto_labeling.yaml")
run_script("bash scripts/prepare_demo_assets.sh /srv/sdg/data/vda_inputs")
```

## Available Scripts

Use script-level `--help` for exact arguments.

| Script | Role |
|---|---|
| `scripts/preflight_credentials.sh` | Secrets/control-plane preflight and workflow image access checks |
| `scripts/pre_submit_guard.py` | Submit-time interpolation, cache, and dataset safety checks |
| `scripts/prepare_demo_assets.sh` | Demo video pull + flatten for default demo path |
| `scripts/generate_configs.py` | Setup-time config and cookbook projection generation |
| `scripts/cosmos_worker.sh` | Augmentation worker execution |
| `scripts/pl_original_worker.sh` | Original-video auto-labeling worker execution |
| `scripts/pl_augmented_worker.sh` | Augmented-video auto-labeling worker execution |
| `scripts/osmo_barrier.py` | Multi-node barrier synchronization |
| `scripts/stage_run_artifacts.sh` | Local mirror of full run output + input video |
| `scripts/render_side_by_side.sh` | Side-by-side comparison render from local artifacts |

## Supported Flows

| Flow | OSMO YAML | Group sequence | Typical use |
|---|---|---|---|
| `augmentation_and_al` | `assets/configs/osmo/augmentation_and_al.yaml` | setup -> augmentation -> auto_labeling_augmented | Augment one or more videos, then auto-label augmented outputs |
| `auto_labeling` | `assets/configs/osmo/auto_labeling.yaml` | setup -> auto_labeling | Label original videos only |
| `e2e` | `assets/configs/osmo/e2e.yaml` | setup -> (auto_labeling_original + augmentation) -> auto_labeling_augmented | Throughput-first path |
| `e2e_super_resolution` | `assets/configs/osmo/e2e_super_resolution.yaml` | setup -> auto_labeling_original -> augmentation -> auto_labeling_augmented | Sequential path with SR gate before augmentation |

Legacy alias `assets/configs/osmo/augmentation_and_pl.yaml` remains for
backwards compatibility.

### Pick the right workflow for the user's request

| User intent | Workflow |
|---|---|
| "Label my source videos" / "PL-only" / "no augmentation" | `auto_labeling` |
| "Create augmented videos and label them" | `augmentation_and_al` |
| "Run the full pipeline quickly" | `e2e` |
| "Run full pipeline, but gate on SR-enhanced originals first" | `e2e_super_resolution` |

## Disambiguation: handle vague requests before committing

Default to autonomy: ask only when missing information blocks execution.

### Autonomous defaults (do NOT ask)

- If dataset source is absent, run VDA demo path (`scripts/prepare_demo_assets.sh`)
  and continue with `dataset=vda-demo`.
- If flow is not explicitly requested, default to `augmentation_and_al`.
- If endpoint mode is unspecified, default to in-cluster persistent NIM reuse and
  automatic NIM deploy/repair when unhealthy.
- If cache is missing, run `setup_model_cache.yaml`, rerun pre-submit guard, and
  continue automatically on success.
- After any stage completes successfully, continue to the next stage immediately.
  Do not pause with "Ready when you are" or equivalent approval prompts.

### Triggers that should pause for disambiguation

| Missing input | Why it matters | Ask |
|---|---|---|
| `USER_INPUT_REQUIRED` from preflight | Required secret is missing | Ask one concise unblock question for exactly the missing value(s) |
| Storage backend prefix cannot be derived from the active dataset/upload root | Wrong scheme causes runtime storage auth mismatch | "What is the backend-native root prefix for this run?" |
| No ONLINE GPU pool/platform can be selected | Workflow cannot schedule setup/workers | "Which GPU pool/platform should this run target?" |

### When NOT to disambiguate

- Do not ask for cookbook unless user explicitly asks to change scene profile.
- Do not offer external endpoints by default.
- Do not ask A/B cache strategy questions; default is automatic cache setup.
- Do not ask to scale down existing NIMs; this is forbidden.
- Do not invent, scrape, or generate random videos when input is missing.
- Do not use non-VDA demo sources (for example Carline adaptation assets) unless
  the user explicitly requests a different dataset.

## Step 0: Select Flow and Gather Inputs

### Input video policy (non-negotiable)

- Always preserve user-provided video inputs (dataset URL, local path, or upload
  folder) as first-class and preferred.
- Never replace an explicit user video with demo assets or any other source.
- If no video input is provided, default to VDA demo assets via
  `scripts/prepare_demo_assets.sh` (HF dataset flow) without asking extra
  source-selection questions.
- If the user explicitly mentions an input video or dataset, prefer and use that
  input instead of demo assets.
- Use only VDA demo assets (`nvidia/video-data-augmentation-demo`) for the
  default demo path.
- Never propose arbitrary web clip downloads or placeholder videos
  unless the user explicitly requests that behavior.

Collect only missing values:

1. Dataset source (prefer explicit user-provided `dataset_url` or local upload
   folder; otherwise default to VDA demo assets and proceed).
2. Flow (`auto_labeling`, `augmentation_and_al`, `e2e`, `e2e_super_resolution`);
   default to `augmentation_and_al` when unspecified.
3. OSMO `gpu_platform` for all VDA resources (auto-select an ONLINE platform