upgrading-cluster-version

What this skill does

# Upgrading Cluster Version

Guides CockroachDB version upgrades end-to-end. Before providing procedures, this skill gathers deployment context to deliver only the steps relevant to the operator's tier.

## When to Use This Skill

- Planning or executing a version upgrade (Self-Hosted)
- Initiating a major version upgrade via Cloud Console (Advanced, BYOC)
- Configuring how patches are applied (Advanced, BYOC)
- Verifying upgrade completion (all tiers)
- Deciding whether to finalize or roll back (Self-Hosted)

**For pre-upgrade health check:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).
**For node drain during Self-Hosted upgrade:** Use [performing-cluster-maintenance](../performing-cluster-maintenance/SKILL.md).
**For maintenance window configuration:** Use [performing-cluster-maintenance](../performing-cluster-maintenance/SKILL.md).

---

## Step 1: Gather Context

### Required Context

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Completely different upgrade procedures per tier |
| **Current version?** | e.g., v24.2.5 | Validates upgrade path and compatibility |

### Additional Context (by tier)

**If Self-Hosted:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Upgrade type?** | Major (e.g., 24.2→24.3), Patch (e.g., 24.2.5→24.2.8) | Major requires finalization; patches do not |
| **Target version?** | e.g., v24.3.1 | Confirms version is available and sequential |
| **Deployment platform?** | Bare metal, VMs, Kubernetes (Operator/Helm/manual) | Changes binary replacement and restart procedure |
| **Process manager?** | systemd, manual, container orchestrator | Changes stop/start commands |
| **Node count?** | Number | Affects upgrade sequencing |

**If Advanced or BYOC:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Upgrade type?** | Major version, Patch | Major requires Console initiation; patches use maintenance windows |
| **Cloud provider?** (BYOC only) | AWS, GCP, Azure | For infrastructure-level monitoring during upgrade |
| **Release channel?** | Regular, Innovation | Innovation gives latest features, shorter support window |

**If Standard or Basic:** No additional context needed — upgrades are fully automatic.

### Context-Driven Routing

| Tier | Go To |
|------|-------|
| Self-Hosted | [Self-Hosted Upgrade](#self-hosted-upgrade) |
| Advanced | [Advanced Upgrade](#advanced-upgrade) |
| BYOC | [BYOC Upgrade](#byoc-upgrade) |
| Standard | [Standard Upgrade](#standard-upgrade) |
| Basic | [Basic Upgrade](#basic-upgrade) |

---

## Self-Hosted Upgrade

**Applies when:** Tier = Self-Hosted

### Major vs Patch

| Aspect | Major (e.g., 24.2→24.3) | Patch (e.g., 24.2.5→24.2.8) |
|--------|--------------------------|------------------------------|
| Finalization required | Yes | No |
| Rollback possible | Before finalization | Always (binary swap) |
| Auto-finalization | Enabled by default (disable recommended) | N/A |

### Pre-Upgrade Validation

```bash
# All nodes live, version-consistent, fully replicated
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>
```

In the output: every node should show `is_live = true`, the `build` column should be a single value, and `ranges_underreplicated` should be `0` everywhere.

```sql
-- No bulk operations running
WITH j AS (SHOW JOBS)
SELECT job_id, job_type, status, now() - created AS running_for FROM j
WHERE status IN ('running', 'paused')
  AND job_type IN ('SCHEMA CHANGE', 'BACKUP', 'RESTORE', 'IMPORT', 'NEW SCHEMA CHANGE');

-- No pending finalization from a previous upgrade
SHOW CLUSTER SETTING cluster.preserve_downgrade_option;
```

### Disable Auto-Finalization (Major Version — Recommended)

```sql
SET CLUSTER SETTING cluster.preserve_downgrade_option = '<current_version>';
-- Example: SET CLUSTER SETTING cluster.preserve_downgrade_option = '24.2';
```

### Rolling Node Upgrade (repeat for each node)

**If process manager = systemd:**
```bash
cockroach node drain --self --certs-dir=<certs-dir> --host=<node-address>
sudo systemctl stop cockroachdb
cp /path/to/new/cockroach /usr/local/bin/cockroach
cockroach version  # verify new binary
sudo systemctl start cockroachdb
```

**If Kubernetes (Operator):**
```bash
kubectl patch crdbcluster <name> --type merge -p '{"spec":{"cockroachDBVersion":"<new-version>"}}'
# Operator handles rolling restart automatically
```

**If Kubernetes (Helm):**
```bash
helm upgrade cockroachdb cockroachdb/cockroachdb --set image.tag=<new-version>
```

**If Kubernetes (manual StatefulSet):**
```bash
kubectl set image statefulset/cockroachdb cockroachdb=cockroachdb/cockroach:<new-version>
```

**Verify each node before proceeding to the next:**
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node> <upgraded-node-id>
```
The targeted node should show `is_live = true` on the new `build`.

### Monitor Progress

```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
Compare the `build` column across all rows. Nodes still on the old version are pending; rolling upgrade is complete when every row shows the new version.

### Finalize (Major Version Only — Irreversible)

Confirm via `cockroach node status` that the `build` column has a single value (every node upgraded). Then:

```sql
RESET CLUSTER SETTING cluster.preserve_downgrade_option;
SHOW CLUSTER SETTING version;  -- Monitor until updated
```

### Roll Back (Before Finalization Only)

Verify `cluster.preserve_downgrade_option` still returns the old version, then replace each node's binary with the previous version and restart. After all nodes are back:
```sql
RESET CLUSTER SETTING cluster.preserve_downgrade_option;
```

---

## Advanced Upgrade

**Applies when:** Tier = Advanced

Advanced clusters are managed by Cockroach Labs. You initiate major upgrades; patches are applied automatically.

### Major Version Upgrade

1. **Cloud Console → Cluster → Settings → Upgrade**
2. Select target version
3. CRL performs rolling upgrade across nodes automatically
4. Monitor progress in Cloud Console
5. Finalize via Cloud Console when testing is complete

**Verification during upgrade:**
```bash
cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
```
Tally the `build` column to see how many nodes are on the new version vs the old.

### Patch Upgrades

Patches are applied automatically during the configured maintenance window. See [performing-cluster-maintenance](../performing-cluster-maintenance/SKILL.md) for maintenance window configuration and patch deferral.

### Release Channel

- **Regular:** Stability-focused, longer support windows
- **Innovation:** Latest features, shorter support, can be skipped

Configure via Cloud Console → Cluster → Settings → Upgrades.

### Upgrade Before End of Support

You are responsible for initiating major upgrades before the current version reaches End of Support (EOS). Failure to upgrade before EOS may affect SLA guarantees.

### Cloud API

```bash
# Check current version
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
  "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>" | jq '.cockroach_version'
```

---

## BYOC Upgrade

**Applies when:** Tier = BYOC

BYOC upgrade procedures are the same as Advanced. Follow all [Advanced Upgrade](#advanced-upgrade) steps.

### Cloud Provider Monitoring During Upgrade

Since BYOC clusters run in your cloud account, you can observe the rolling upgrade in your infrastructure:

**If AWS:** EC2 console shows instance restarts; CloudWatch metrics show brief dips during node cycling.

**If GCP:** Compute Engine console shows VM restarts; Cloud Monitoring shows instance-level events.

**If Azure:** Azure portal shows VM restarts; Azure Monitor captures instance events.

### Additional BYOC Considerations

- Verify PrivateLink/PSC/VPC Peering connections