hyperpod-ssm

What this skill does

# HyperPod SSM Access

## Prerequisites

- **`aws` CLI v2**, authenticated for the target account/Region.
- **`session-manager-plugin`** — installed alongside the AWS CLI.
- **`jq`** — the scripts build JSON payloads with it.
- **`unbuffer`** (from the `expect` package) — wraps `aws ssm start-session` with a PTY so the session-manager-plugin flushes stdout instead of racing to close. Without it, calls intermittently return empty output with `Cannot perform start session: EOF` even when the command ran. Install with `sudo yum install expect`, `sudo apt install expect`, or `brew install expect`. `ssm-exec.sh` detects and uses it automatically; falls back with a warning if missing.

## SSM Target Format

Target: `sagemaker-cluster:<CLUSTER_ID>_<GROUP_NAME>-<INSTANCE_ID>`

- `CLUSTER_ID`: Last segment of cluster ARN (NOT the cluster name). Extract via `get-cluster-info.sh`.
- `GROUP_NAME`: Instance group name — retrieve via `list-nodes.sh`.
- `INSTANCE_ID`: EC2 instance ID (e.g., `i-0123456789abcdef0`)

## Scripts

Three scripts under `scripts/`. Resolve cluster info and nodes **once**, then execute per node.

### get-cluster-info.sh — Resolve cluster name → ID (call once)

```bash
scripts/get-cluster-info.sh CLUSTER_NAME [--region REGION]
# Output: {"cluster_id":"...","cluster_arn":"...","cluster_name":"...","region":"..."}
```

### list-nodes.sh — List all nodes with pagination (call once)

```bash
scripts/list-nodes.sh CLUSTER_NAME [--region REGION] [--instance-group GROUP] [--instance-id ID]
# Output: JSON array of ClusterNodeSummaries (InstanceId, InstanceGroupName, InstanceStatus, etc.)
```

`list-cluster-nodes` paginates at 100 nodes. This script handles pagination automatically.

### ssm-exec.sh — Execute command on a node (call per node)

```bash
# Execute — with pre-built target
scripts/ssm-exec.sh --target "sagemaker-cluster:CLUSTERID_GROUP-INSTANCEID" 'command' [--region REGION]

# Execute — with parts
scripts/ssm-exec.sh --cluster-id ID --group GROUP --instance-id INSTANCE_ID 'command' [--region REGION]

# Upload
scripts/ssm-exec.sh --target TARGET --upload LOCAL_PATH REMOTE_PATH [--region REGION]

# Read remote file
scripts/ssm-exec.sh --target TARGET --read REMOTE_PATH [--region REGION]
```

## Running Commands Across Many Nodes

SSM `start-session` rate limit: **3 TPS** per account. Plan batch size and delay accordingly.

`aws ssm send-command` does NOT support `sagemaker-cluster:` targets — only `start-session` works.

## Manual SSM Commands

When the scripts aren't suitable, use `aws ssm start-session` directly with `AWS-StartNonInteractiveCommand`. Wrap every invocation in `unbuffer` — without it, stdout is intermittently empty (see Prerequisites).

```bash
cat > /tmp/cmd.json << 'EOF'
{"command": ["bash -c 'echo hello && whoami'"]}
EOF

unbuffer aws ssm start-session \
  --target sagemaker-cluster:{CLUSTER_ID}_{GROUP_NAME}-{INSTANCE_ID} \
  --region REGION \
  --document-name AWS-StartNonInteractiveCommand \
  --parameters file:///tmp/cmd.json
```

- Always use a JSON file for `--parameters` — inline parameters break with special characters.
- The document's `command` parameter is argv, not shell input. Wrap multi-statement scripts in `bash -c '...'` so pipes, semicolons, and redirects evaluate.

## Common Diagnostic Commands

| Task             | Command                                                        |
| ---------------- | -------------------------------------------------------------- |
| Lifecycle logs   | `cat /var/log/provision/provisioning.log`                      |
| Memory           | `free -h`                                                      |
| Disk/mounts      | `df -h && lsblk`                                               |
| GPU status       | `nvidia-smi`                                                   |
| GPU memory       | `nvidia-smi --query-gpu=memory.used,memory.total --format=csv` |
| EFA/network      | `fi_info -p efa`                                               |
| CloudWatch agent | `sudo systemctl status amazon-cloudwatch-agent`                |
| Top processes    | `ps aux --sort=-%mem \| head -20`                              |

## Key Details

- Default SSM non-interactive user is `root`.
- SSM rate limit: **3 TPS** per account.
- For interactive sessions (rare), omit `--document-name` to get a shell.
- Interactive commands (vim, top) are not supported via `AWS-StartNonInteractiveCommand`.
- Large outputs may be truncated by SSM.
- For troubleshooting common errors, see [references/troubleshooting.md](references/troubleshooting.md).