alibabacloud-ecs-reboot-or-crash-diagnosis

What this skill does

# ECS Instance Reboot/Crash Diagnosis

Diagnose root cause of ECS instance unexpected reboot or crash. Uses standard workflow: check platform maintenance events first, then check internal system logs. Supports both Linux and Windows systems.

## Required Parameters

Before starting diagnosis, **must** obtain the following parameters from user:

| Parameter | Description | Example |
|------|------|------|
| `INSTANCE_ID` | ECS instance ID | `i-bp1a2b3c4d5e6f7g8h9j` |
| `REGION_ID` | Region ID | `cn-hangzhou` |

**If user does not provide any of the above parameters, must ask user first. Do not start diagnosis.**

## Mandatory Execution Rules

1. **Must obtain parameters first** — Instance ID and Region ID are required. Must ask user if missing.
2. **Standard workflow cannot be skipped** — Must execute in order: Maintenance Event Check → OSType Detection → System Log Check
3. **Must check Cloud Assistant status before diagnostics** — Before executing Step 3A/3B, must verify Cloud Assistant is running via `DescribeCloudAssistantStatus`. If not running, provide alternative diagnostic approaches.
4. **All diagnostic conclusions must be based on actual data** — No fabrication, speculation, or assumptions
5. **Output format must be strictly followed** — After diagnosis, **must read the complete template in `references/output-format.md`**, output strictly according to template structure. No free-form output, no omitted sections, no changed hierarchy. Every placeholder `{...}` in the template must be filled with actual data.

---

## Prerequisites

### CLI Tools
- **aliyun-cli 3.3.3+** (required) — For calling Alibaba Cloud API
- Installation & configuration: see [CLI Installation Guide](../../cli-installation-guide.md)

### AI-Mode Configuration (Required)
Before using aliyun CLI commands, must configure AI-Mode:

```bash
# Enable AI-Mode
aliyun configure ai-mode enable

# Set user-agent for skill identification
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-ecs-reboot-or-crash-diagnosis"

# Update plugins
aliyun plugin update
```

**After diagnosis complete, disable AI-Mode:**
```bash
aliyun configure ai-mode disable
```

### Alibaba Cloud Credentials
Credentials must be pre-configured **outside of agent session**. Agent only verifies:
```bash
aliyun configure list
```

### Instance Requirements
- **Cloud Assistant client must be installed and running** on the instance
  - Alibaba Cloud Linux: Pre-installed by default
  - Ubuntu/CentOS/Other: May require manual installation, check with `DescribeCloudAssistantStatus` API
  - Installation guide: https://help.aliyun.com/document_detail/64930.html
- Instance status must be Running
- **Note:** If Cloud Assistant is not running, diagnostic commands cannot be executed remotely. Must provide manual diagnostic steps to user.

---

## Required RAM Permissions

See **[RAM Policies](references/ram-policies.md)** for the complete permission list and custom policy example.

---

## Step 1: Confirm Instance Information (Cannot Skip)

**Verify instance exists and get basic information:**

```bash
aliyun ecs describe-instances \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-ids '["<INSTANCE_ID>"]'
```

Confirm from returned JSON:
- `RegionId` — Region ID (matches user provided)
- `Status` — Instance status (Running/Stopped)
- `InstanceName` — Instance name
- `OSType` — Operating system type (**windows / linux**)

**Record OSType for Step 3 branch selection.**

---

## Step 2: Check ECS Maintenance Events

**Query instance historical system events to determine if platform maintenance caused reboot:**

```bash
aliyun ecs describe-instance-history-events \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID> \
  --event-cycle-status Executed
```

**Event Analysis:**

| Event Type | Meaning | Determination | Next Step |
|---|---|---|---|
| `SystemMaintenance.Reboot` | Reboot caused by system maintenance | Platform-initiated maintenance | Inform user, no further investigation needed |
| `SystemFailure.Reboot` | Reboot caused by underlying hardware/system failure | Platform infrastructure failure | Suggest instance migration or contact support |
| `InstanceFailure.Reboot` | Reboot caused by instance-level failure | **Instance internal issue detected by platform** | **Must continue to Step 3 for system log check** |
| `InstanceExpiration.Stop` | Instance stopped due to expiration | Billing issue | Need renewal, no further investigation |
| No relevant events | No platform maintenance events found | Not platform-initiated | Continue to Step 3 |

**Important Notes for InstanceFailure.Reboot:**
- This event indicates the platform detected an instance-level anomaly and triggered automatic recovery
- Common causes: kernel panic, OOM, system hang, critical process failure
- **Must execute Step 3** to check system logs for root cause
- Even if no obvious errors in logs, the instance may have been unresponsive at kernel level

**If maintenance event found:**
- Clearly inform user of reboot cause (event type, time, reason)
- Provide handling suggestions
- End diagnosis flow

**If no maintenance event found:**
- Continue to Step 3, check internal system logs based on OSType

---

## Step 3A: Linux System Diagnosis (Execute when OSType is linux)

### Step 3A.1: Check Cloud Assistant Status (Mandatory)

**Before executing diagnostic commands, verify Cloud Assistant is running:**

```bash
aliyun ecs describe-cloud-assistant-status \
  --biz-region-id <REGION_ID> \
  --region <REGION_ID> \
  --instance-id <INSTANCE_ID>
```

**Check the response:**
```json
{
  "InstanceCloudAssistantStatusSet": {
    "InstanceCloudAssistantStatus": [
      {
        "InstanceId": "i-xxx",
        "RegionId": "cn-xxx",
        "CloudAssistantStatus": "true",
        "LastHeartbeatTime": "2026-04-09T07:26:58Z"
      }
    ]
  }
}
```

**Important Notes:**
- `CloudAssistantStatus` is a **string** ("true"/"false"), not boolean
- Check `LastHeartbeatTime` to ensure it's recent (within last few minutes)
- Even if status is "true", RunCommand may still fail if service is unstable
- **Always check RunCommand execution result** and handle failures gracefully
- **Ubuntu vs RHEL differences:**
  - RHEL/CentOS/Alibaba Cloud Linux: Service name is `kdump`, crash files named `vmcore-*`
  - Ubuntu/Debian: Service name is `kdump-tools`, crash files named `dump.*` and `dmesg.*`
  - Diagnostic script now checks both service names and all crash file types

**If CloudAssistantStatus is false or command fails:**
- Cloud Assistant is not installed or not running on the instance
- **Cannot proceed with remote diagnostic commands**
- **Alternative approaches:**
  1. Guide user to SSH into the instance and check logs manually
  2. Provide manual diagnostic commands for user to execute
  3. Suggest installing Cloud Assistant: [Installation Guide](https://help.aliyun.com/document_detail/64930.html)
  4. Check instance monitoring data via CloudMonitor API

**If CloudAssistantStatus is true:**
- Proceed to Step 3A.2

### Step 3A.2: Execute Linux Diagnostic Script

Execute Linux diagnostic script via Cloud Assistant to check:
- System reboot records (`last reboot`, `/var/log/messages` or `/var/log/syslog`)
- Kernel Panic records (`dmesg`)
- OOM records and `vm.panic_on_oom` configuration
- Kdump configuration and crash dump file status
- **Crash dump files**: vmcore (RHEL/CentOS) or dump.*/dmesg.* (Ubuntu/Debian)

**Complete diagnostic commands: see [diagnostic-commands.md](references/diagnostic-commands.md#linux-system-diagnosis)**

**Linux Result Analysis:**

| Finding | Possible Cause | Suggestion |
|---|---|---|
| Kernel Panic + crash dump (vmcore/dump.*) | Kernel crash, dump file generated | Read dmesg.* file for panic reason, contact Alibaba Cloud technical support for deep analysis |
| Kernel Panic + no crash dump | Kernel crash, but kdump not configured or not working | **Procee