rag-perf

What this skill does

# RAG-Perf — config-driven perf benchmark CLI

## Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side **profiling pass** (per-stage timing, citation quality, bottleneck inference) and an optional **aiperf load test** (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: `rag-perf -c <config>` plus `--help` / `--version`. Behaviour is *fully* config-driven; field variations belong in YAML.

## Scope

- **Accuracy / RAGAS** scoring of answer quality → use the **rag-eval** skill.
- **Deploying, repairing, or configuring services** (compose, helm, NIM env vars) → use the **rag-blueprint** skill.
- **Production monitoring / alerting** — rag-perf is a one-shot benchmark tool.
- **Runtime requirement:** a deployed RAG server reachable on the network.

## Prerequisites

- Repo cloned; **run commands from the repo root** (config paths in the presets are repo-root-relative).
- Python **3.11+** and **uv** on PATH.
- Install rag-perf into its own uv-managed venv: `uv sync --project scripts/rag-perf`.
- For unit tests: install dev extras as well — `uv sync --project scripts/rag-perf --extra dev` (otherwise `pytest-asyncio` is missing and async tests error out at collection time).
- A reachable RAG server (default `http://localhost:8081`). For the aiperf phase, the bundled `nvidia_rag` endpoint plugin must be installed — `pip install -e ./scripts/rag-perf` registers it via the `aiperf.plugins` entry point.
- For **synthetic** queries: an OpenAI-compatible chat-completions endpoint reachable at `synthetic.llm_url` (default `http://localhost:8999/v1/chat/completions`).
- rag-perf itself runs without `NVIDIA_API_KEY` (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

## Instructions

1. **Pick a preset.** The three under [`scripts/rag-perf/configs/`](../../scripts/rag-perf/configs) are:
   - `quick_profile.yaml` — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
   - `single_run.yaml` — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
   - `sweep.yaml` — multi-axis sweep. `load.concurrency`, `rag.vdb_top_k`, `rag.reranker_top_k` are all `int | list[int]`; any of them as a list becomes a sweep axis (Cartesian product).

2. **Edit the preset.** **Required:** replace `rag.collection_names: ["<collection_name>"]` with a real collection on the deployed ingestor server. Verify the collection exists via `GET /v1/collections` on the ingestor. The placeholder `<collection_name>` validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.

3. **Run.** From repo root:
   ```bash
   uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
   ```
   Same form for the other presets. The CLI accepts only `-c / --config` (required), `--help`, `--version`.

4. **Read stdout.** Every invocation prints, in order: a startup banner, a one-line summary, the **fully resolved config as YAML** (so the run is reproducible from terminal output), per-grid-point progress with the **shlex-joined aiperf command** in copy-pastable form, a **rich per-point summary table** (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a **side-by-side comparison table** auto-labelled by whichever axis varied. See [`references/output-and-analysis.md`](references/output-and-analysis.md).

5. **Inspect artifacts.** Layout depends on run shape — flat for single-point + `iterations=1`, nested under `iter_<i>/<point>/...` otherwise. See [`references/output-and-analysis.md`](references/output-and-analysis.md) for the full directory tree, file purposes, and how to parse `results.json` / `results.csv` / `report.md`.

6. **Summarise for the user.** When reporting back, follow the playbook in [`references/output-and-analysis.md#summarising-results-to-the-user`](references/output-and-analysis.md#summarising-results-to-the-user): pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, **always flag** zero citations / non-zero error rate / suspect `llm_ttft_ms` / small-sample p99, and propose a concrete next-experiment YAML.

7. **Tune.** Schema is fully documented in [`docs/performance-benchmarking.md`](../../docs/performance-benchmarking.md) and the deeper-dive references below. Common knobs: turn `aiperf.enabled: false` for profile-only mode, increase `load.iterations` for variance estimation, set `load.sleep_between_points_s: 60` for overnight Cartesian sweeps.

## Examples

**Profile-only (quickest signal on retrieval / reranker tuning):**

```bash
uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/quick_profile.yaml
```

Output: `rag-perf-results/quick_profile/run_<ts>/{profile_report.md, profile_results.json, profiling/}`. The `aiperf_rag_on/` directory is omitted. Filenames are `profile_*` because `aiperf.enabled: false`.

**Single benchmark point with full report:**

```bash
uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
```

Output: flat `run_<ts>/{report.md, results.json, results.csv, profiling/, aiperf_rag_on/}`.

**Concurrency sweep:**

```bash
uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/sweep.yaml
```

Output: nested `run_<ts>/iter_1/<CR:_VDB-K:_RERANKER-K:_…>/{profiling,aiperf_rag_on}/` per point, plus aggregate `report.md` / `results.json` / `results.csv` at the run root.

**Run unit tests:**

```bash
uv sync --project scripts/rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/rag-perf python -m pytest tests/unit/test_rag_perf/
```

## Limitations

- The CLI is **config-only**: author or copy YAML to vary a parameter.
- `load.concurrency` / `rag.vdb_top_k` / `rag.reranker_top_k` accept `int | list[int]`; the validator requires unique list values because each value names a unique point dir.
- `input.file` and `input.synthetic` follow an XOR rule — both set fails validation. When neither is set, `synthetic` auto-fills with defaults so a bare config still validates.
- File-based input format is **inferred from extension only** (`.jsonl` or `.csv`); other extensions are rejected.
- Synthetic generation streams each query to disk as it completes (failure-resilient) but **fails fast on the first LLM error** — partial JSONL is preserved. Re-run after fixing the endpoint.
- Reasoning models (Nemotron Omni, Qwen-Reasoning) require `synthetic.disable_thinking: true` (the default). Without it the model exhausts the token budget on chain-of-thought and `content` returns empty — the generator now raises with a clear message instead of substituting `reasoning_content` for the answer.
- aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing `AiperfRunner._base_aiperf_cmd` in `scripts/rag-perf/rag_perf/runner.py`.
- Procedural detail lives under **`references/`** to keep this file concise.

## Troubleshooting

| Error / signal | Likely cause | What to do |
|---|---|---|
| `Configuration errors in <yaml>:  •  input  —  ... XOR rule` | Both `input.file` and `input.synthetic` set | Pick one. The XOR validator runs at YAML load time. |
| `input.file must end in .jsonl or .csv` | Extension other than `.jsonl` / `.csv` | Rename or convert. |
| `load.concurrency has duplicate values` | e.g. `[2, 2, 4]` | Each concurrency maps to a unique point dir; dedupe. |
| `warmup_requests must be >= 1` | YAML had `warmup_requests: 0` | aiperf rejects warmup=0; minimum is 1. |
| `LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).` | Reasoning mod