Claude
Skills
Sign in
Back

alibabacloud-aes-sysom-pai-diagnosis

Included with Lifetime
$97 forever

Perform SysOM deep diagnosis on Alibaba Cloud PAI products (EAS / DLC) to identify root causes of instance-level issues. Use when users report: - EAS instance anomalies, GPU OOM (out of memory), GPU memory out-of-bounds errors - Slow first-token latency, uneven request scheduling across model service instances - OOM (Out Of Memory), insufficient memory, processes being killed - Abnormally high system load, high IO latency, network jitter, packet loss - Instance crashes, unexpected restarts, kernel oops - DLC training job hangs, communication timeouts, per-step throughput degradation - Any issue related to EAS instance health, DLC job health, or underlying compute resource performance

Cloud & DevOps

What this skill does


# alibabacloud-aes-sysom-pai-diagnosis

> **Skill Name**: alibabacloud-aes-sysom-pai-diagnosis
> **Goal**: Perform SysOM deep diagnosis on Alibaba Cloud PAI products (EAS / DLC) to identify root causes of instance-level performance and health issues.

---

## Credential Security

> **[CRITICAL] Credential Security Rules:**
> - **NEVER** print, echo, or display AccessKey ID / AccessKey Secret values in conversation or command output (even partial masking of `LTAI_ACCESS_KEY_ID` is FORBIDDEN)
> - **NEVER** ask the user to input AK/SK directly in the conversation or command line
> - **NEVER** use `aliyun configure set` with literal credential values
> - **ONLY** use `aliyun configure list` to check credential status
>
> ```bash
> aliyun configure list
> ```
> Check the output for a valid profile (AK, STS, or OAuth identity).
>
> **If no valid profile exists, STOP here.**
> 1. Obtain credentials from [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak)
> 2. Configure credentials **outside of this session** (via `aliyun configure` in terminal or environment variables in shell profile)
> 3. Return and re-run after `aliyun configure list` shows a valid profile

---

## RAM Policy

For the full list of RAM permissions required by this skill, see [references/ram-policies.md](references/ram-policies.md).

> **[MUST] Permission Failure Handling:** When any command or API call fails due to permission errors at any point during execution, follow this process:
> 1. Read `references/ram-policies.md` to get the full list of permissions required by this SKILL
> 2. Use `ram-permission-diagnose` skill to guide the user through requesting the necessary permissions
> 3. Pause and wait until the user confirms that the required permissions have been granted

---

## Parameter Confirmation

> **IMPORTANT: Parameter Confirmation** — Before executing any command or API call,
> ALL user-customizable parameters (e.g., RegionId, instance IDs, product type,
> time ranges, etc.) MUST be confirmed with the user. Do NOT assume or use default
> values without explicit user approval.

| Parameter | Required/Optional | Description | Default Value |
|-----------|-------------------|-------------|---------------|
| `region` | Required | Region of the PAI resource (e.g., `cn-hangzhou`) | None, must be provided by user |
| `instance` | Required | PAI instance ID (EAS service ID `eas-m-xxx` or DLC job ID `dlcxxxxxxxx`) | None, must be provided by user |
| `product` | Required | PAI sub-product type, one of `EAS` or `DLC` | Auto-inferred from `instance` prefix (`eas-` → `EAS`, `dlc` → `DLC`); only ask user when inference fails |
| `start_time` | Optional | Diagnosis start timestamp (Unix seconds) | `0` (real-time) |
| `end_time` | Optional | Diagnosis end timestamp (Unix seconds) | `0` |
| `enable_diagnosis` | Optional | Force real-time diagnosis (highest priority) | `false` |
| `uid` | Optional | Account ID owning the resource | `None` |
| `ocd_description` | Optional | User's problem description in English, with words joined by underscores (`_`). No Chinese characters, no spaces. Example: `GPU_OOM_instance_restart` | `None` |

### Product Auto-Inference Rule

The `product` field MUST be present in the params JSON. The value is determined as follows:

1. If the user explicitly specifies `product` (`EAS` or `DLC`), use the user value
2. Otherwise, infer from the `instance` prefix:
   - `eas-` → `EAS`
   - `dlc` (no hyphen, e.g., `dlcxxxxxxxx`) → `DLC`
3. If inference is ambiguous or fails, you **MUST** explicitly ask the user to choose between `EAS` and `DLC`

---

## Core Workflow

The workflow has two phases with 8 steps. All `aliyun` CLI **business commands** (SysOM, EAS, DLC API calls) **MUST** include `--user-agent AlibabaCloud-Agent-Skills/alibabacloud-aes-sysom-pai-diagnosis`. System commands (`version`, `configure`, `plugin`) do NOT use `--user-agent`.

### Phase 1: Environment Setup (Steps 0–3)

**Step 0 — Enable AI-Mode and Update Plugins**

Before executing any CLI commands, enable AI-Mode, set User-Agent, and update plugins:

```bash
aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-aes-sysom-pai-diagnosis"
aliyun plugin update
```

> **⚠️ The above three commands must be executed before all CLI operations, and only need to be run once.**

**Step 1 — CLI Version Check**

```bash
aliyun version
```

Verify version >= 3.3.1. If not met, refer to `references/cli-installation-guide.md` for installation.

**Step 2 — Enable Auto Plugin Installation**

```bash
aliyun configure set --auto-plugin-install true
```

**Step 3 — Credential Verification**

```bash
aliyun configure list
```

If no valid credentials exist, **STOP** and guide the user to configure credentials outside the session.

---

### Phase 2: Diagnosis Execution (Steps 4–8)

For detailed workflow, see [references/diagnose-workflow.md](references/diagnose-workflow.md).

**Step 4 — Ambiguous Problem Clarification (Inversion Gate)**

Must confirm `region`, `instance`, and **when the anomaly occurred**. If not provided by the user, ask explicitly. `product` is auto-inferred from the `instance` prefix (`eas-` → `EAS`, `dlc` → `DLC`); only ask user when inference fails. Also extract optional time range.

> **⚠️ Time Inference Rule**: When the user's description contains **any temporal reference** (e.g., "this morning", "yesterday afternoon", "around 3pm", "last night"), you **MUST** proactively ask for the specific time range and recommend **historical diagnosis mode**. Do NOT silently default to real-time diagnosis when the problem clearly occurred in the past.

**Step 5 — SysOM Role Initialization**

```bash
aliyun sysom initial-sysom --check-only false --source aes-skills --user-agent AlibabaCloud-Agent-Skills/alibabacloud-aes-sysom-pai-diagnosis
```

**Step 6 — Resource Validation**

Before invoking diagnosis, you **MUST** validate the resource based on the inferred `product`:

#### 6A. EAS — Verify Service Exists

```bash
aliyun eas list-services \
  --region <region> \
  --filter <eas_service_id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-aes-sysom-pai-diagnosis
```

From the returned `Services` array, verify that an entry with a matching `ServiceId` exists. If no match is found, inform the user that the service ID is invalid and stop the pipeline.

#### 6B. DLC — Verify Resource Type is Lingjun

```bash
aliyun pai-dlc get-job \
  --region <region> \
  --job-id <dlc_job_id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-aes-sysom-pai-diagnosis
```

Check the `ResourceType` field in the response:
- `Lingjun` → proceed to Step 7
- Any other value → **STOP** and inform the user: "SysOM diagnosis currently only supports DLC jobs running on Lingjun resources. Your job uses `<ResourceType>`, which is not yet supported."

> **⚠️ The `instance` field in params JSON uses the original instance ID directly (`eas-m-xxx` or `dlcxxxxxxxx`) — this step is purely for validation.**

**Step 7 — Invoke Diagnosis and Poll Results**

#### Diagnosis Mode Decision Rules

```
if enable_diagnosis == true:
    mode = real-time diagnosis    # enable_diagnosis has highest priority
elif start_time != 0:
    mode = historical diagnosis   # time range specified, retrospective analysis
else:
    mode = real-time diagnosis    # default
```

- **Real-time**: `start_time=0`, `end_time=0`
- **Historical**: `start_time=<unix_ts>`, `end_time=<unix_ts>`
- **Forced real-time**: when `enable_diagnosis=true`, force `start_time` to 0 even if provided

#### Build params JSON

Use **snake_case** keys (consistent with SDK). Required base fields (**ALL must be included**):

```json
{
  "instance": "<eas_service_id_or_dlc_job_id>",
  "region": "<region>",
  "product": "<EAS_or_DLC>",
  "start_time": 0,
  "end_time": 0,
  "type": "ocd",
  "ai_roadmap": true,
  "enable_sysom_link": false,
  "ocd_description": "<user_problem_description_in_english_with_underscore

Related in Cloud & DevOps