alibabacloud-pai-dlc-job-diagnostics

What this skill does

# PAI-DLC Job Diagnostics and Health Inspection

Read-only diagnostic analysis for PAI-DLC distributed training jobs, covering
three scenarios:

- **Queuing-stuck root cause analysis** — quota check, node scheduling
  diagnosis, hyper-node availability
- **Failed-job localization** — failure classification, logs/events evidence
  chain, root cause identification
- **Cluster health inspection** — training throughput, hang detection,
  SanityCheck, restart stability

**Architecture**: PAI-DLC Job (read-only queries) + PAI Studio Resource
Diagnosis API (queuing scenario).

## 0. Dependencies

This skill performs **read-only diagnostics** only. All write operations
(create / update / stop jobs, resource discovery, etc.) live in the companion
skill `alibabacloud-pai-dlc-job`. The two skills are complementary in
responsibility and share a common field contract.

| Prerequisite Skill | Role | When to switch to it |
|--------------------|------|----------------------|
| `alibabacloud-pai-dlc-job` | Write ops (create/update/stop) + AIWorkSpace resource discovery | Creating / modifying / stopping jobs, or discovering Image / Dataset / CodeSource |
| This skill | Read-only diagnostics (logs / events / sanity-check / queuing root cause) | Job already exists — troubleshooting or health inspection |

**Discover and install the prerequisite skill:**

```bash
# Discover available skills
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-find-skills
# Install the alibabacloud-pai-dlc-job skill itself
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-pai-dlc-job
```

**Cross-skill field contract:** The `--job-id` / `--pod-id` values this skill
consumes are produced verbatim by `alibabacloud-pai-dlc-job` via
`list-jobs` / `get-job --cli-query "Pods[0].PodId"` — no transformation needed.
`--region` / `--workspace-id` follow the same resolution rules in both skills.

## Installation Requirements

> **Pre-check: Aliyun CLI >= 3.3.1 required**
> Run `aliyun version` to verify >= 3.3.1. If not installed or version too low,
> see [references/cli-installation-guide.md](references/cli-installation-guide.md).
> Then [MUST] run `aliyun configure set --auto-plugin-install true`.

> **Note on `--user-agent`:** Every API-invoking `aliyun` command in this skill MUST
> include `--user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics`. Client-side helpers
> (`aliyun version`, `aliyun configure ...`, `aliyun plugin ...`,
> `aliyun <product> --help`) do not invoke remote APIs and therefore do not require
> the flag.

```bash
aliyun version
aliyun configure set --auto-plugin-install true
aliyun plugin update
aliyun pai-dlc --help
aliyun paistudio --help >/dev/null 2>&1 || aliyun plugin install --names aliyun-cli-paistudio

aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics"
# After session: aliyun configure ai-mode disable
```

## Authentication

> **Pre-check: Alibaba Cloud Credentials Required**
>
> **Security Rules:**
> - **NEVER** read, echo, or print AK/SK values
> - **NEVER** ask the user to input AK/SK directly
> - **ONLY** use `aliyun configure list` to check credential status
>
> ```bash
> aliyun configure list
> ```
> Check the output for a valid profile (AK, STS, or OAuth identity).
>
> **If no valid profile exists, STOP here.**
> 1. Obtain credentials from [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak)
> 2. Configure credentials **outside of this session**
> 3. Return and re-run after `aliyun configure list` shows a valid profile

## RAM Permissions

> **[MUST] Permission Failure Handling:** When any command fails due to permission errors:
> 1. Read `references/ram-policies.md` for the full permission list
> 2. Use `ram-permission-diagnose` skill to guide the user
> 3. Pause and wait until the user confirms permissions have been granted

| Product | Permissions | Purpose |
|---------|-------------|---------|
| pai-dlc | `pai:GetJob`, `pai:GetPodLogs`, `pai:GetJobEvents`, `pai:GetPodEvents`, `pai:ListJobSanityCheckResults` | Job information collection |
| paistudio | `paistudio:GetQuotaWorkloadDiagnosis` | Queuing resource diagnosis |

## Parameter Confirmation

> **IMPORTANT: Parameter Confirmation** — Before executing any command,
> ALL user-customizable parameters (RegionId, JobId, etc.) MUST be confirmed with the user.

| Parameter | Required | Description |
|-----------|----------|-------------|
| `region` | Yes | Region where the job runs |
| `job_id` | Yes | DLC job ID (e.g., `dlcXXX`) |

---

## Entry Routing

When a diagnostic request arrives, first call `get-job` to fetch job status,
then route by status:

| Job status | Route to scenario |
|------------|-------------------|
| `Queuing` / `Creating` | → Queuing-stuck root cause analysis |
| `Failed` | → Failed-job localization |
| `Running` | → Health inspection |
| `Stopped` | Inform the user "job was actively stopped", no diagnosis |
| `Succeeded` | → Historical review (follow Scenario 3 Execution steps) |

**Edge case — job was queuing but is now Stopped/Succeeded**: If the user
describes the job as "stuck in queue" but `get-job` shows `Stopped` or
`Succeeded`, still route to Scenario 1 (queuing analysis) but expect the
resource diagnosis API to return HTTP 400. Follow the "Fallback on API Failure"
procedure in Scenario 1.

Users may also directly request a specific scenario (e.g., "run a health
inspection" even when status is not Running).

---

## Diagnostic Toolbox

### PAI-DLC Read-Only Commands

```bash
aliyun pai-dlc get-job --region <r> --job-id <id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-job-events --region <r> --job-id <id> --max-events-num 50 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-pod-events --region <r> --job-id <id> --pod-id <pod> --max-events-num 20 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-pod-logs --region <r> --job-id <id> --pod-id <pod> --max-lines 100 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc list-job-sanity-check-results --region <r> --job-id <id> \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
aliyun pai-dlc get-job-sanity-check-result --region <r> --job-id <id> --sanity-check-number 1 \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
```

### PAI Studio Resource Diagnosis (queuing scenario only)

```bash
aliyun paistudio GET /api/v1/quotas/{quota_id}/workloads/{job_id}/diagnosis \
  --region <r> --header "Content-Type=application/json" --force \
  --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-dlc-job-diagnostics
```

**Hard constraint**: `quota_id` MUST come from `get-job`'s `ResourceId` field.
If `ResourceId` is empty (public pay-as-you-go), this API is unavailable.

Full API structure: see [references/resource-diagnosis-api.md](references/resource-diagnosis-api.md).

---

## Scenario 1: Queuing-Stuck Root Cause Analysis

**Trigger**: job status = `Queuing` / `Creating` and user reports it cannot be scheduled.

**Tools**: `get-job` → `paistudio resource diagnosis` → (optional) `get-job-events`.

**Hard constraints**:
- `ResourceId` empty → resource diagnosis unavailable; mine events for clues
- `ResourceId` non-empty → resource diagnosis API is the primary instrument

> **CRITICAL: pai-dlc vs paistudio — two different products**
>
> | Product | Scope | Commands |
> |---------|-------|----------|
> | **pai-dlc** | Job/Pod lifecycle (GetJob, GetJobEvents, ListJobs, GetPodLogs) | `aliyun pai-dlc get-job ...` |
> | **paistudio** | Platform-level services including resource diagnosis | `aliyun paistudio GET /api/v1/quotas/...` |
>
> The resource diagnosis API belongs to **paistudio**, NOT pai-dlc.
> Do **NOT** call `pai-dlc Get