managing-cluster-capacity

What this skill does

# Managing Cluster Capacity

Manages cluster capacity across all CockroachDB deployment tiers. What "capacity" means varies by tier — Self-Hosted manages individual nodes, Advanced/BYOC manage node count and machine size, Standard manages provisioned vCPUs, and Basic auto-scales with cost controls.

## When to Use This Skill

- Permanently removing a node from a cluster (Self-Hosted)
- Adding nodes to increase capacity (Self-Hosted)
- Scaling cluster node count or machine size (Advanced, BYOC)
- Adjusting provisioned compute (Standard)
- Managing costs on a serverless cluster (Basic)
- Replacing hardware or migrating infrastructure (Self-Hosted, BYOC)
- Replacing a failed or dead node (Self-Hosted)
- Managing storage utilization and disk pressure (Self-Hosted)

**For temporary maintenance (not capacity changes):** Use [performing-cluster-maintenance](../performing-cluster-maintenance/SKILL.md).
**For pre-operation health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).

---

## Step 1: Gather Context

### Required Context

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Different capacity model per tier |
| **Direction?** | Scale up (add capacity), Scale down (reduce capacity) | Determines procedure |

### Additional Context (by tier)

**If Self-Hosted (scaling down):**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **How many nodes to remove?** | 1, multiple | Multi-node decommission should be done simultaneously |
| **Target node IDs?** | Node IDs from `cockroach node status` | Required for CLI commands |
| **Is the node alive or dead?** | Alive, Dead | Dead nodes use a different procedure |
| **Deployment platform?** | Bare metal, VMs, Kubernetes | Changes CLI and cleanup steps |
| **Current replication factor?** | 3, 5, custom | Must have enough nodes remaining |
| **Current node count?** | Number | Validates remaining capacity |
| **Storage utilization?** | Low (<60%), Medium (60-80%), High (>80%) | Determines urgency and whether storage maintenance is needed |

**If Advanced or BYOC:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Scale method?** | Cloud Console, API, Terraform | Determines procedure |
| **Current and target configuration?** | e.g., 5 nodes → 3 nodes, or 4 vCPU → 8 vCPU | Validates constraints |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | Affects infrastructure verification |

**If Standard:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Current provisioned vCPUs?** | Number | Context for scaling decision |
| **Target vCPUs?** | Number | Validates workload will fit |

**If Basic:** Gather cost management goals — Basic auto-scales with no manual capacity control.

### Context-Driven Routing

| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Capacity Management](#self-hosted-capacity-management) |
| Advanced | [Advanced Scaling](#advanced-scaling) |
| BYOC | [BYOC Scaling](#byoc-scaling) |
| Standard | [Standard Compute Management](#standard-compute-management) |
| Basic | [Basic Cost Management](#basic-cost-management) |

---

## Self-Hosted Capacity Management

**Applies when:** Tier = Self-Hosted

### Scaling Down: Decommission Nodes

#### Pre-Decommission Validation

```bash
# All nodes live, version-consistent, with replication and per-node range counts
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

Inspect the output for:
- `is_live = true` for every node
- `ranges_underreplicated` is `0` everywhere (all ranges fully replicated)

```sql
-- Replication factor (and other zone-level settings)
SHOW ZONE CONFIGURATION FOR RANGE default;
```

For per-store capacity (so you can verify remaining nodes won't exceed 60% utilization after absorbing the decommissioned node's data), use the DB Console **Overview** → **Storage** page or scrape the Prometheus metrics endpoint:

```bash
curl -ks https://<node>:8080/_status/vars | grep '^capacity'
```

Node count after decommission must be ≥ the zone's `num_replicas`.

#### If Node Is Alive: Drain Then Decommission

```bash
# Step 1: Drain
cockroach node drain <node_id> --certs-dir=<certs-dir> --host=<any-live-node>

# Step 2: Decommission (single node)
cockroach node decommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>

# Step 2: Decommission (multiple nodes — more efficient, do simultaneously)
cockroach node decommission <id_1> <id_2> <id_3> --certs-dir=<certs-dir> --host=<any-live-node>
```

#### If Node Is Dead: Replace Failed Node

When a node has been dead longer than `server.time_until_store_dead` (default 5m), CockroachDB automatically re-replicates its data to surviving nodes. Use this procedure to clean up the dead node and optionally add a replacement.

**Step 1: Confirm the node is dead and data is safe**

```bash
# Confirm the dead node and verify replication has caught up
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

In the output: the dead node should show `is_live = false`, and every surviving node should show `ranges_underreplicated = 0`. For per-store capacity on the surviving nodes, use the DB Console **Overview** → **Storage** page.

If under-replicated ranges exist, wait for re-replication to complete before proceeding.

**Step 2: Decommission the dead node (metadata cleanup)**

```bash
cockroach node decommission <dead_node_id> --certs-dir=<certs-dir> --host=<any-live-node>
```

**Step 3: Add a replacement node (recommended)**

If remaining nodes are above 60% utilization, provision a replacement node using the [Scaling Up: Add Nodes](#scaling-up-add-nodes) procedure.

**Multiple dead nodes:** Decommission all dead nodes simultaneously:
```bash
cockroach node decommission <id_1> <id_2> --certs-dir=<certs-dir> --host=<any-live-node>
```

See [replacing-failed-nodes reference](references/replacing-failed-nodes.md) for detailed failure scenarios and recovery procedures.

#### Monitor Decommission Progress

```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

Wait for `gossiped_replicas = 0` and `membership = 'decommissioned'`. Then stop the process on the decommissioned node.

#### Cancel a Decommission

```bash
cockroach node recommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>
```

Only works while still in `decommissioning` state.

### Scaling Up: Add Nodes

1. Provision new hardware/VM with same specs as existing nodes
2. Install same CockroachDB version (`cockroach version` to confirm)
3. Start node with `--join` pointing to existing cluster nodes
4. Verify join and monitor rebalancing:
   ```bash
   cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
   ```
   The new node should appear in the output with `is_live = true`. The `ranges` column climbs as data rebalances toward the new node.

### Post-Scaling Verification

```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

Expect `ranges_underreplicated = 0` on every node and a balanced `ranges` count across nodes. For per-store capacity utilization, use the DB Console **Overview** → **Storage** page.

---

## Advanced Scaling

**Applies when:** Tier = Advanced

Advanced clusters are managed by Cockroach Labs. Capacity is adjusted by changing node count or machine size.

### Via Cloud Console

1. **Cluster → Capacity**
2. Adjust node count or machine type (vCPUs per node)
3. CRL handles all node operations (drain, decommission, provisioning) safely
4. Monitor progress in Cloud Console

### Via Cloud API

```bash
# Scale node count
curl -X PATCH -H "Authorization: Bearer $COCKROACH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"config": {"num_nodes": <new_count>}}' \
  "https://cockroachlabs.cloud/api/v1/clusters/<cl