hyperpod-slurm-debugger

What this skill does

# HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on
HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.
For remediation, link to the official AWS or Slurm documentation.

## When to invoke

Invoke when the user reports any of the symptoms in the [decision table](#decision-table).

## When NOT to invoke

- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.
- NCCL training-hang investigation — invoke `hyperpod-nccl`.
- Node unreachable via SSM — invoke `hyperpod-ssm`.

## Constraints

- Read-only. Do not run, recommend, or print state-mutating commands.
- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation
  drifts the live state from the IaC plan.

Canonical recovery URLs:
[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).

## Prerequisites

- AWS CLI v2, authenticated for the target account and region with permissions:
  - `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
  - `ssm:StartSession` on the HyperPod-created SSM document
- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
  installed locally.
- `jq` ≥ 1.6.
- `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`
  returns empty stdout intermittently with `Cannot perform start session: EOF` and every
  check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian /
  Ubuntu / macOS. Script exits at prerequisite check if missing.

## Procedure

### Step 1 — Collect inputs

Ask the user for:

1. HyperPod cluster name (not Slurm partition name).
2. AWS region.
3. Optional: a specific Slurm node name.

### Step 2 — Confirm orchestrator

```bash
aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
  --query 'Orchestrator' --output json
```

If `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).

### Step 3 — Run the diagnostic script

```bash
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
# Scope to a node:
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>
```

Relay the script output to the user verbatim.

### Step 4 — Map findings → docs

For each finding, look up the section in the [decision table](#decision-table) and link
the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.

## Decision table

| Symptom (`sinfo -o "%N %T %30E"` or script finding)         | Section                                                |
| ----------------------------------------------------------- | ------------------------------------------------------ |
| Node state = `down` or `down*`, reason other than below     | [A: Node Down](#a-node-down)                           |
| Node state = `down*`, Reason = `Node unexpectedly rebooted` | [B: Unexpected Reboot](#b-unexpected-reboot)           |
| Jobs `PENDING` with `REASON=Resources` while nodes are idle | [C: Controller State](#c-controller-state)             |
| Jobs stuck `COMPLETING` after node replacement              | [C: Controller State](#c-controller-state)             |
| `scontrol ping` returns `DOWN` for the controller           | [C: Controller State](#c-controller-state)             |
| GRES (GPU) counts incorrect or not released                 | [C: Controller State](#c-controller-state)             |
| `state=fail` issued but no recovery occurred                | [D: Action Reason Mismatch](#d-action-reason-mismatch) |
| Accounting errors or RPC errors mentioning `dbd`            | [C: Controller State](#c-controller-state) (slurmdbd)  |
| `slurm.conf` edited; new partitions or nodes not visible    | [C: Controller State](#c-controller-state) (config)    |
| Job exited on a hardware failure but did not restart        | [E: Auto-resume](#e-auto-resume)                       |

## Defaults

| Behavior             | Default                                                                                            | Override                   |
| -------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- |
| Mode                 | read-only — always; no remediation flag exists                                                     | n/a                        |
| Region               | `$AWS_DEFAULT_REGION`, falling back to `us-east-1`                                                 | `--region <R>`             |
| Scope                | all nodes in `down` / `drain` / `fail` / "unexpectedly rebooted"                                   | `--node <SLURM_NODE_NAME>` |
| Output               | colorized terminal                                                                                 | `--no-color`               |
| SSM target format    | `sagemaker-cluster:<clusterId>_<instanceGroupName>-<instanceId>` (derived)                         | n/a                        |
| Controller discovery | `--controller-group` (if set) → `SlurmConfig.NodeType=Controller` → `provisioning_parameters.json` | `--controller-group <N>`   |

## Error handling

| Failure                                            | Skill behavior                         | Required user action                            |
| -------------------------------------------------- | -------------------------------------- | ----------------------------------------------- |
| `describe-cluster` fails                           | Print AWS error; exit 1                | Fix credentials/region; verify cluster name     |
| Cluster has `Orchestrator.Eks`                     | Exit 1 with pointer to EKS-side skills | Use `hyperpod-node-debugger` or `hyperpod-nccl` |
| `session-manager-plugin` missing / SSM unreachable | `sinfo` returns empty; exit 1          | Install plugin; verify node `InService`         |
| Disk ≥ 95 % full on a `down` node                  | Report finding `disk-full-<node>`      | Refer to AWS troubleshooting docs               |
| Missing `jq` or `aws`                              | Exit 1 at prerequisite check           | Install per [Prerequisites](#prerequisites)     |

---

## A: Node Down

Node is `down` because `slurmd` stopped responding. Causes: `slurmd` crash, disk full,
OOM, network partition, hardware fault.

Script checks: `systemctl is-active slurmd`, `srun -w <NODE> hostname` (RPC layer), disk,
memory.

Link: <https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md>

If node returns to `down` after a manual resume → escalate to `hyperpod-node-debugger`.

Context: [references/slurm-details.md § A](references/slurm-details.md#-a-node-down--diagnostic-context).

---

## B: Unexpected Reboot

Node is `down*` with Reason `"Node unexpectedly rebooted"` because `slurmd`
re-registered after an out-of-band reboot. Upstream Slurm behavior, not HyperPod.
Node is typically healthy.

Links:

- <https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md>
- <https://slurm.schedmd.com/scontrol.html> (`state=resume` semantics)

If node reboots again within minutes → escalate to `hyperpod-node-debugger`.

Context: [references/slurm-details.md § B](references/slurm-details.md#-b-unexpected-reboot--diagnostic-context).

---

## C: Controller State

`slurmctld` in-memory state can desync from the on-disk state. A controller restart reloads from `StateSaveLocation` and clears bad caches. User decides and executes.

Restart may help:

| Symptom                                            | Why