performing-cluster-maintenance

What this skill does

# Performing Cluster Maintenance

Manages planned cluster maintenance across all deployment tiers. For Self-Hosted, this means draining and restarting individual nodes. For Advanced/BYOC, this means configuring and managing maintenance windows for CRL-applied patches. For Standard and Basic, maintenance is fully managed with no customer action required.

## When to Use This Skill

- Planning OS patching, hardware changes, or configuration updates (Self-Hosted)
- Configuring or modifying a maintenance window (Advanced, BYOC)
- Setting patch deferral policies (Advanced, BYOC)
- Monitoring during a CRL-managed maintenance event (Advanced, BYOC)
- Running pre-maintenance validation checks (Self-Hosted, Advanced, BYOC)
- Understanding how maintenance affects your application (all tiers)
- Preparing applications for maintenance events (all tiers)

**For permanent node removal:** Use [managing-cluster-capacity](../managing-cluster-capacity/SKILL.md).
**For pre-maintenance health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
**For version upgrades:** Use [upgrading-cluster-version](../upgrading-cluster-version/SKILL.md).

---

## Step 1: Gather Context

### Required Context

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines maintenance procedure |
| **Goal?** | Plan maintenance, Configure maintenance window, Defer a patch, Monitor during maintenance, Prepare application | Routes to the right procedure |

### Additional Context (by tier)

**If Self-Hosted:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Maintenance type?** | OS patching, Hardware change, Binary upgrade, Config change, Planned restart | Affects sequencing and post-maintenance steps |
| **Deployment platform?** | Bare metal, VMs, Kubernetes (Operator/Helm/manual) | Changes drain and restart commands |
| **Process manager?** | systemd, manual, container orchestrator | Changes stop/start commands |
| **Target node ID?** | Node ID | Required for drain command |
| **Long-running queries expected?** | Yes (increase drain timeout), No (default timeout) | Determines drain-wait parameter |

**If Advanced or BYOC:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Maintenance window configured?** | Yes (what schedule), No | Determines if window needs setup |
| **Patch pending?** | Yes, No, Don't know | Determines urgency |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring |

**If Standard or Basic:** No context needed — maintenance is fully managed.

### Context-Driven Routing

| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Node Maintenance](#self-hosted-node-maintenance) |
| Advanced | [Advanced Maintenance Management](#advanced-maintenance-management) |
| BYOC | [BYOC Maintenance Management](#byoc-maintenance-management) |
| Standard | [Standard Maintenance](#standard-maintenance) |
| Basic | [Basic Maintenance](#basic-maintenance) |

---

## Self-Hosted Node Maintenance

**Applies when:** Tier = Self-Hosted

Self-Hosted operators manage all maintenance directly. The core operation is draining a node to safely move leases and connections before stopping it.

### Pre-Maintenance Checks

Run all checks before any maintenance operation. **Stop if any check fails.**

**Checks 1-3, 5 (node liveness, drain state, replication, version consistency):**

```bash
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

Stop conditions in the output:
- any `is_live = false` (Check 1)
- any `is_draining = true` (Check 2)
- any `ranges_underreplicated > 0` (Check 3)
- multiple distinct values in the `build` column (Check 5)

**Check 4: No disruptive jobs running (WAIT or pause before proceeding):**

```sql
WITH j AS (SHOW JOBS)
SELECT job_id, job_type, status, now() - created AS running_for FROM j
WHERE status IN ('running', 'paused')
  AND job_type IN ('SCHEMA CHANGE', 'BACKUP', 'RESTORE', 'IMPORT', 'NEW SCHEMA CHANGE');
```

**Check 6: Storage utilization safe (WARNING if any node > 70%):**

No production-safe SQL view exposes per-store capacity. Use the DB Console **Overview** → **Storage** page or scrape the per-node Prometheus endpoint:

```bash
curl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'
```

**Stop conditions:** Do not proceed with maintenance if any node is not live, ranges are under-replicated, another node is draining, or a rolling upgrade is in progress. Wait for running jobs to complete or pause them.

See [maintenance-prechecks reference](references/maintenance-prechecks.md) for a consolidated precheck script.

### Execute Drain

**If platform = bare metal or VMs:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address>
```

**If long-running queries expected:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address> --drain-wait=60s
```

**If platform = Kubernetes:**
```bash
# Operator handles drain automatically during pod eviction
kubectl delete pod <pod-name>
# Or for rolling restart:
kubectl rollout restart statefulset cockroachdb
```

### Stop, Maintain, Restart

**If process manager = systemd:**
```bash
sudo systemctl stop cockroachdb
# ... perform maintenance ...
sudo systemctl start cockroachdb
```

**If process manager = manual:**
```bash
kill -TERM $(pgrep -f 'cockroach start')
# ... perform maintenance ...
cockroach start --certs-dir=<certs-dir> --store=<path> --join=<addresses> --background
```

Never use `kill -9` unless the process is unresponsive to SIGTERM.

### Post-Restart Verification

```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```

The restarted node should show `is_live = true`. The `replicas_leaseholders` column for that node should increase over the next several minutes as leases rebalance back.

See [drain-details reference](references/drain-details.md) for drain phases, timeout configuration, and advanced monitoring.

### Storage Maintenance

Periodic storage maintenance for Self-Hosted clusters:

**Ballast file verification:**
```bash
ls -lh <store-path>/auxiliary/EMERGENCY_BALLAST
# If missing, create: cockroach debug ballast <store-path>/auxiliary/EMERGENCY_BALLAST --size=1GiB
```

**Disk utilization check:**

Use the DB Console **Overview** → **Storage** page or the per-node Prometheus endpoint:

```bash
curl -ks https://<node>:8080/_status/vars | grep -E '^capacity( |_used|_available)'
```

Nodes above 70% utilization should be addressed before maintenance — draining a node temporarily increases load on remaining nodes.

---

## Advanced Maintenance Management

**Applies when:** Tier = Advanced

Advanced clusters are managed by Cockroach Labs. CRL applies patches and performs infrastructure maintenance during the configured maintenance window. You do not drain or restart nodes — CRL handles this using rolling restarts.

### Configure a Maintenance Window

1. **Cloud Console → Cluster → Settings → Maintenance**
2. Set a weekly 6-hour window
   - Choose day of week (e.g., Sunday)
   - Choose start time in UTC (e.g., 02:00 UTC)
   - Window duration is 6 hours

If no window is configured, CRL applies patches at a time of their choosing.

### View Current Maintenance Window

**Cloud Console → Cluster → Settings → Maintenance** shows the current schedule.

**Cloud API:**
```bash
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
  "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.maintenance_window'
```

### Defer Patches

If a pending patch needs to be delayed (e.g., for testing):

1. **Cloud Console → Cluster → Settings → Upgrades**
2. Select deferral period: **30, 60, or 90 days**

Deferred patches still apply at the end of the deferral period. Deferral only delays — it does not skip.

### What Happens During Maintenance

1