alibabacloud-emr-starrocks-manage

What this skill does

# Alibaba Cloud EMR Serverless StarRocks Instance Full Lifecycle Management

Manage StarRocks instances via the `aliyun` CLI. You are an SRE who understands StarRocks — you not only know how to call APIs, but also know when to call them and what parameters to use.

## Authentication

Reuse the profile already configured in the `aliyun` CLI. Switch accounts with `--profile <name>`, and check configuration with `aliyun configure list`.

## Domain Knowledge

### Product Overview

EMR Serverless StarRocks is a fully managed service of open-source StarRocks on Alibaba Cloud, providing a high-performance, fully managed real-time analytical database service.

**Core Features**:

- **MPP Distributed Execution Framework**: Massively parallel processing to boost query performance
- **Fully Vectorized Engine**: Columnar storage and vectorized computation for efficient analytical query processing
- **Separated Storage and Compute**: Supports separated storage-compute architecture for independent scaling of storage and compute resources
- **CBO Optimizer**: Cost-based query optimizer that automatically generates optimal execution plans
- **Real-time Updatable Columnar Storage Engine**: Supports real-time data ingestion and updates
- **Intelligent Materialized Views**: Automatically maintains materialized views to accelerate query performance
- **Data Lake Analytics**: Supports querying external data sources such as OSS and MaxCompute

### Use Cases

- **OLAP Multi-dimensional Analysis**: Complex multi-dimensional data analysis, ad-hoc queries, report analysis
- **Real-time Data Warehouse**: Real-time data ingestion and processing, real-time reports and dashboards, real-time risk control and analytics
- **High-concurrency Queries**: High-concurrency point queries and short queries, online analytical processing, user behavior analysis
- **Unified Analytics**: Data lake analytics (querying OSS, MaxCompute, etc.), lakehouse architecture, cross-datasource federated queries

### Core Concepts

| Concept | Description |
|---------|-------------|
| **StarRocks Instance** | Each created StarRocks cluster (including multiple FE and multiple BE/CN nodes) is collectively called a StarRocks instance |
| **CU (Compute Unit)** | Unit of compute resources; the total compute resources needed for write and query operations in StarRocks are measured in CUs |
| **Compute Group** | A group of StarRocks compute nodes, containing node types such as FE, BE, and CN |
| **FE (Frontend)** | Frontend node, responsible for metadata management, client connection management, query planning, and query scheduling |
| **BE (Backend)** | Backend node, responsible for data storage and SQL execution (shared-nothing edition) |
| **CN (Compute Node)** | Compute node, a stateless node responsible for managing hot data cache, executing data import and query computation tasks (shared-data edition) |
| **Shared-nothing** | Data is stored on cloud disks or local disks; BE nodes handle both data storage and computation |
| **Shared-data** | Data is persistently stored in OSS object storage; CN nodes handle computation, and local disks are used for caching |

**FE Node Roles**:

- **Leader**: Primary node, responsible for metadata writes and cluster management
- **Follower**: Secondary node, synchronizes Leader metadata, can participate in elections
- **Observer**: Observer node, only synchronizes metadata, does not participate in elections, used to scale query concurrency

### Instance Types

When creating an instance, you need to choose the architecture type:

| Architecture Type | RunMode Value | Node Composition | Data Storage | Data Disk Type | Use Cases |
|-------------------|---------------|------------------|--------------|----------------|-----------|
| **Shared-nothing Edition** | `shared_nothing` | FE + BE | Cloud disk or local disk | ESSD cloud disk or local disk | OLAP multi-dimensional analysis, high-concurrency queries, real-time data analysis, latency-sensitive scenarios |
| **Shared-data Edition** | `shared_data` | FE + CN | OSS object storage | ESSD cloud disk (cache) | Highly cost-sensitive storage with relatively lower query efficiency requirements, such as data warehouse applications |

**Shared-nothing Architecture Features**:

- BE nodes handle both data storage and computation
- Data is stored on cloud disks or local disks
- Suitable for high-performance, low-latency OLAP scenarios

**Shared-data Architecture Features**:

- Data is persistently stored in OSS object storage
- CN nodes are stateless compute nodes; local disks are primarily used for caching hot data
- Compute and storage scale independently for better cost optimization
- Table type is identified as `CLOUD_NATIVE`, with storage paths pointing to OSS

### Compute Resource Specifications (CU)

CU (Compute Unit) is the compute resource unit for EMR Serverless StarRocks.

**CU Specification Types**:

| Spec Type | SpecType Value | Features | Use Cases |
|-----------|---------------|----------|-----------|
| **Standard** | `standard` | Balanced compute and memory configuration | General OLAP analysis |
| **Memory Enhanced** | `ramEnhanced` | Higher memory ratio | Complex queries, high concurrency |
| **Network Enhanced** | `networkEnhanced` | Higher network bandwidth | External table analysis with large data scan volumes |
| **High-performance Storage** | `localSSD` | High-performance storage access | High I/O scenarios with strict storage I/O performance requirements |
| **Large-scale Storage** | `bigData` | Large capacity storage | Extremely large data volumes, cost-sensitive |

> **Note**: The SpecType for FE node groups only supports `standard`. The multiple spec types above only apply to BE/CN node groups.

### Storage Specifications

| Storage Type | Performance Level | Max IOPS | Max Throughput | Use Cases |
|-------------|-------------------|----------|----------------|-----------|
| ESSD PL0 | Entry-level | 10,000 | 180 MB/s | Development and testing |
| ESSD PL1 | Standard | 50,000 | 350 MB/s | General production |
| ESSD PL2 | High-performance | 100,000 | 750 MB/s | High-performance requirements |
| ESSD PL3 | Ultra-performance | 1,000,000 | 4,000 MB/s | Ultra-performance requirements |

### Billing Methods

**Billing Items**:

| Billing Item | Description | Billing Method |
|-------------|-------------|----------------|
| Compute Resources (CU) | Compute resources for FE and BE/CN nodes | Subscription / Pay-as-you-go |
| Storage Resources | Cloud disks, elastic temporary disks, data storage | Billed by actual usage |
| Backup Storage | Storage space occupied by data backups | Billed by actual usage |

**Payment Methods**:

| Payment Method | API Parameter Value (PayType) | Description |
|---------------|-------------------------------|-------------|
| **Pay-as-you-go** | `postPaid` | Pay after use, billing generated hourly, suitable for short-term needs/testing |
| **Subscription** | `prePaid` | Pay before use, suitable for long-term needs, more cost-effective |

**Payment Method Conversion**:

- Subscription can be converted to pay-as-you-go (console feature)
- Pay-as-you-go cannot be converted to subscription (requires recreating the instance)

**Cost Components**:

**Shared-nothing Edition Costs**:

- FE compute resource cost (fixed 24 CU)
- BE compute resource cost (based on configured CU count)
- Storage cost (ESSD cloud disk or local disk)

**Shared-data Edition Costs**:

- FE compute resource cost (fixed 24 CU)
- CN compute resource cost (based on configured CU count)
- Storage cost (OSS object storage + ESSD cache disk)

### Version Series

| Version Series | PackageType Value | Features | Use Cases | Spec Support | Region Restrictions |
|---------------|-------------------|----------|-----------|--------------|---------------------|
| **Standard Edition** | `official` | Full functionality, production-grade stability, supports all spec types | Production environments, core business | Supports standard, m