launch-nemo-rl

What this skill does

# launch-nemo-rl — running NeMo-RL recipes on Kubernetes via nrl-k8s

This is the playbook for the `nrl-k8s` CLI at `infra/nrl_k8s/`. Follow it when the user asks to launch / iterate / debug a NeMo-RL recipe on a Kubernetes cluster. Verify current state (`kubectl`, `git log`, the recipe + infra files) before acting — the cluster is shared and the cost of a wrong action is high.

## 1. One command, two modes

There is a single top-level submission command: **`nrl-k8s run`**. It has two lifecycle modes.

| Mode               | Invocation        | When to use                                                                                                                                                                   | Cluster after? |
| :----------------- | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |
| Ephemeral (default)  | `nrl-k8s run`              | One-shot. KubeRay applies a RayJob, runs, tears the cluster down. Best for most runs.                                                                                          | No (auto)      |
| Long-lived           | `nrl-k8s run --raycluster` | Dev loop. Reuses a matching live cluster, applies if absent, warns + reuses on drift (pass `--recreate` to replace). Then submits daemons and training. First-choice for iteration. | Yes            |

Ask: *Do I need this cluster after the run?* If yes, use `--raycluster`. Otherwise use the default (ephemeral).

The rest of the CLI is observability / stage-by-stage control:

| Command                 | Purpose                                                                                         |
| :---------------------- | :---------------------------------------------------------------------------------------------- |
| `nrl-k8s check`         | Validate a recipe + infra pair; optionally write the fully-resolved manifests (`-o`).           |
| `nrl-k8s status`        | Per-role RayCluster state, head pod phase, worker pod phases, daemon job status.                |
| `nrl-k8s cluster up/down/list/dashboard` | Manage RayClusters independently of a run (e.g. render a manifest with `--dry-run`). |
| `nrl-k8s job list/logs/stop` | Observability over Ray Jobs already submitted to a role's cluster.                         |
| `nrl-k8s logs`          | Tail a role's pod / daemon logs without needing a submission id.                                |

## 2. Recipe + infra pair

Every launch takes two files. Pass the infra with `--infra`, not merged inline:

```
nrl-k8s run infra/nrl_k8s/examples/<recipe>.yaml \
  --infra infra/nrl_k8s/examples/<recipe>.<profile>.infra.yaml
```

- **Recipe** (e.g. `qwen3_30b_math_8n_4gpu.yaml`) — NeMo-RL config: model, GRPO/SFT knobs, `cluster.{gpus_per_node,num_nodes}`. Uses `defaults:` to inherit from `examples/configs/recipes/llm/...`.
- **Infra** (e.g. `*.<profile>.infra.yaml`) — K8s/Ray shape: namespace, image, service account, RayCluster spec under `kuberay:`, optional Deployments under `deployments:`, `submit.submitter`, `launch.{mode,codeSource,codePath,entrypoint}`. Pair names follow `<recipe>.<profile>[.prod].infra.yaml` where `<profile>` names the hardware target (e.g. `gb300`).

Example pairs in `infra/nrl_k8s/examples/` — read the neighbouring files to see the current conventions for the target profile.

## 3. Long-lived mode flags

Three independent dimensions. `--mode` is a macro that picks defaults; individual flags override it.

```
--mode interactive   → --submitter portForward  --code-source upload  (tails logs)
--mode batch         → --submitter exec         --code-source image   (returns after nohup)
```

- **Submitter**: `portForward` uses `kubectl port-forward` + Ray Job SDK (gets a `submission_id` the dashboard tracks). `exec` uses `kubectl exec` + `nohup` on the head pod (no submission_id; driver appears as `type=DRIVER` in the dashboard).
- **Code source**: `upload` stages a working_dir from the laptop (Ray 100 MiB cap). `image` / `lustre` expect code on the pod's filesystem — paired with `--code-path` (typically `/opt/nemo-rl`), which is a subPath of the shared-filesystem PVC mount in the standard infra examples.
- **Wait**: `--wait` tails logs until terminal; `--no-wait` returns as soon as the driver is running.

Other long-lived-only flags:

- `--replace` — stop any running training / daemon job before submitting new ones (suffixes daemon submissionIds with a timestamp so Ray accepts the resubmit).
- `--recreate` — delete + re-apply a RayCluster whose live spec has drifted from the rendered manifest (default is warn + reuse).
- `--skip-daemons` — bring up all declared clusters but only submit training. Use on disagg recipes where gym/generation are already healthy.

Gotcha: on infra where the entrypoint does `cd /opt/nemo-rl` (or another in-image / Lustre path) and loads the recipe from there, **`--code-source upload` does NOT override the recipe on the pod** — the uploaded working_dir sits in `/tmp/ray/...` but the entrypoint `cd`s away from it. To actually test a local recipe change, either sync your edits to the shared filesystem mounted into the pods or flip the Hydra overrides in the entrypoint.

## 4. Ephemeral mode flags (`--rayjob`)

When `--rayjob` is set, `run` branches into the RayJob code path. Relevant flags:

- `--rayjob-name NAME` — RayJob metadata name (defaults to the training cluster name).
- `--shutdown / --no-shutdown` — default `true`: KubeRay deletes the RayCluster once the Ray Job reaches a terminal state.
- `--ttl SECONDS` — default 3600s: keep the RayJob object around after the run finishes for post-mortem log access.
- `--wait / --no-wait` — default `wait`: poll `jobDeploymentStatus` until Complete/Failed. `--no-wait` returns as soon as the RayJob is applied.
- `--timeout SECONDS` — default 86400s (24h): bound the `--wait` poll.
- `--dry-run` — render the RayJob manifest and print it; do not apply.

`--replace` / `--recreate` / `--skip-daemons` are silently ignored in `--rayjob` mode (KubeRay owns lifecycle).

## 5. Iterating on a config without touching the shared filesystem

When the recipe on the pod filesystem has the wrong value for your experiment, use Hydra overrides on the entrypoint instead of forking the recipe. Pattern:

```yaml
entrypoint: |
  set -eu
  cd /opt/nemo-rl
  RUN_ID="\${RAY_JOB_SUBMISSION_ID:-\${NRL_K8S_RUN_ID:-$(date -u +%Y%m%d-%H%M%S)}}"
  python -u examples/run_grpo.py \
    --config infra/nrl_k8s/examples/<recipe>.yaml \
    logger.wandb_enabled=true \
    logger.wandb.project=<project> \
    "logger.wandb.name=<run-name>-\${RUN_ID}"
```

**Escape `${…}`** with a backslash. OmegaConf otherwise interprets it as interpolation and errors on shell-style `${VAR:-default}`. `RUN_ID` resolves to `RAY_JOB_SUBMISSION_ID` (injected by KubeRay in rayjob mode) → `NRL_K8S_RUN_ID` (injected by the CLI in long-lived mode) → local timestamp — so the name is unique across either path.

## 6. Per-profile concerns (hardware + scheduler + DRA)

Every infra YAML encodes a hardware/scheduler profile. The concrete examples in `infra/nrl_k8s/examples/` are authoritative for the profiles they target — read the neighbouring infra file before writing a new one. Things that commonly vary:

- **Per-node GPUs** (e.g. 4 vs 8) — must match `cluster.gpus_per_node` in the recipe, otherwise workers stay `Pending`.
- **Node selectors** — head pods usually land on a CPU-only node pool; GPU workers match on `nvidia.com/gpu.product` or a node-group label.
- **Scheduler** — KAI (`schedulerName: kai-scheduler` + `kai.scheduler/queue` label) with topology annotations (`kai.scheduler/topology`, `kai.scheduler/topology-required-placement`) gang-schedules workers into one clique. Without it, pods may land on different racks and NVLink/RoCE won't span them.
- **DRA claims** — ComputeDomain + RoCE are attached via `resourceClaim