hyperpod-performance-debugger

What this skill does

# HyperPod Performance Debugger

1. **Uneven NCCL performance across nodes** — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
2. **Poor filesystem performance** — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.

## Scope and delegation

Route findings outside the two in-scope scenarios to the owner skill below.

| Concern observed                                                       | Route to                                                     |
| ---------------------------------------------------------------------- | ------------------------------------------------------------ |
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace  | `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) |
| `Cannot allocate memory` at `os.fork()`, root volume exhausted         | `hyperpod-node-debugger` (§ I Resource Exhaustion)           |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | `hyperpod-nccl`                                              |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes           | `hyperpod-version-checker`                                   |
| EFA self-referencing security-group rule missing — single node         | `hyperpod-node-debugger` § A (EFA / Security Group)          |
| EFA self-referencing security-group rule missing — cluster-wide        | `hyperpod-cluster-debugger` § A (EFA Health Checks)          |
| Slurm node state changes (drain / resume / reboot)                     | `hyperpod-slurm-debugger`                                    |
| Diagnostic bundle for AWS Support                                      | `hyperpod-issue-report`                                      |
| Shell access on a node                                                 | `hyperpod-ssm`                                               |

## Operating policy

- Read-only. Print commands the customer runs; do not execute commands that modify state.
- Container vs host version comparisons go through `hyperpod-version-checker`.
- Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in `hyperpod-node-debugger` § G.

## Workflow

1. Confirm the symptom is **uneven NCCL** or **poor filesystem performance**. If neither, route to the matching sibling skill above.
2. Run `scripts/perf-snapshot.sh` (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.
3. For each `[CONCERN]` line in the script output, open the matching section below and read the supporting reference.
4. After the per-incident diagnosis, recommend the HyperPod platform health features in [§ Continuous health coverage](#continuous-health-coverage) so the customer gets ongoing protection.

## Step 1: Run the snapshot

```bash
bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Scope to one suspect node:
bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>
```

The script samples one node by default. It collects host-side data via `hyperpod-ssm`: `nvidia-smi` output (temperature, SM clocks, PCIe link width, ECC, NVLink, `topo -m`), recent `dmesg` Xid lines, EFA port state and `fi_info` provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, `df -h` / `lfs df -h` per mount, host iowait, `/dev/shm` size, and root-volume usage. All read-only.

Tags: `[OK]` healthy · `[CONCERN]` signal worth investigating (carries a `→` pointer to the owner skill) · `[INFO]` informational.

**Host vs container scope.** The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of `unknown` is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via `hyperpod-version-checker` before drawing conclusions.

## Step 2: Match signal → section

| Observation                                                                   | Section                                                                                                                                |
| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Pairwise NCCL bandwidth varies across node pairs / suspected straggler        | **[A: Uneven NCCL Performance](#a-uneven-nccl-performance)**                                                                           |
| Nodes spread across AZs / network-node-layer labels / UltraServer boundaries  | **[A](#a-uneven-nccl-performance)**                                                                                                    |
| EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible | **[A](#a-uneven-nccl-performance)** + route to `hyperpod-node-debugger` § A; `hyperpod-version-checker` for cross-node version compare |
| `iostat` shows high iowait, FSx CloudWatch utilization sustained near 100%    | **[B: Poor Filesystem Performance](#b-poor-filesystem-performance)**                                                                   |
| DataLoader stalls, checkpoint dominates step time                             | **[B](#b-poor-filesystem-performance)**                                                                                                |
| Xid line in `dmesg`, uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°C      | Route to `hyperpod-node-debugger` § G                                                                                                  |
| Container vs host version drift suspected                                     | Route to `hyperpod-version-checker`                                                                                                    |
| `Cannot allocate memory` at `os.fork()`, root volume full, OOM events         | Route to `hyperpod-node-debugger` § I                                                                                                  |
| NCCL timeout, hang, TCP fallback (`NET/OFI Using TCP`), RDMA memlock          | Route to `hyperpod-nccl`                                                                                                               |

---

## A: Uneven NCCL Performance

The customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.

Per the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.

### Diagnostic pass (read-only)

The host-side data points — GPU thermal/ECC/PCIe/clocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA/OFI/driver versions, `nvidia-smi topo -m` — are all collected by `scripts/perf-snapshot.sh` (Step 1 above). The script tags `[CONCERN]` with thresholds and emits routing pointers; rerun it per suspect node via `--node <INSTANCE_ID>`.

For driver / CUDA / NCCL / EFA / OFI version drift across nodes, run `hyperpod-version-checker` skill.

### Pairwise NCCL bandwidth test

Run the standard `nccl-tests` recipes from [awslabs/awsome-distributed-training](https://github.com/awslabs/awsome-distributed-training/tree/main/micro-benchmarks/nccl-tests). For an N-node cluster, run all-reduce across every pair and record `busbw` for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script fla