managing-certificates-and-encryption

What this skill does

# Managing Certificates and Encryption

Manages TLS certificate and encryption key lifecycle across all deployment tiers. Before providing procedures, this skill gathers context to determine whether the operator manages certificates directly (Self-Hosted), manages CMEK encryption keys (Advanced/BYOC), or has fully managed encryption (Standard/Basic).

## When to Use This Skill

- Monitoring certificate expiration (Self-Hosted)
- Performing scheduled certificate rotation (Self-Hosted)
- Managing CMEK encryption keys (Advanced/BYOC)
- Responding to key compromise (Self-Hosted, CMEK)
- Auditing encryption posture for compliance (all tiers)
- Adding DNS names or IPs to node certificates (Self-Hosted)

**For daily health checks:** Use [reviewing-cluster-health](../reviewing-cluster-health/SKILL.md).

---

## Step 1: Gather Context

### Required Context

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Deployment tier?** | Self-Hosted, Advanced, BYOC, Standard, Basic | Determines encryption management responsibility |
| **Reason?** | Routine monitoring, Scheduled rotation, Key compromise, Compliance audit, Add SAN entries | Determines urgency and procedure |

### Additional Context (by tier)

**If Self-Hosted:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Certificate type?** | CA, Node, Client, UI | Different rotation procedures per type |
| **Deployment platform?** | Bare metal/VMs, Kubernetes (Operator/Helm/manual) | Changes rotation tooling |
| **Certificate tooling?** | cockroach cert, openssl, HashiCorp Vault, cert-manager | Determines generation commands |
| **Is the CA being rotated?** | Yes, No | CA rotation requires combined CA approach |

**If Advanced or BYOC:**

| Question | Options | Why It Matters |
|----------|---------|----------------|
| **Is this about CMEK?** | Yes, No | CMEK is the customer's encryption responsibility; TLS is managed by CRL |
| **Cloud provider?** | AWS, GCP, Azure | Determines KMS service and CLI commands |
| **CMEK currently enabled?** | Yes, No | CMEK must be enabled at cluster creation |

**If Standard or Basic:** No context needed — TLS and encryption are fully managed. CMEK is not available on these tiers.

### Context-Driven Routing

| Tier + Scenario | Go To |
|-----------------|-------|
| Self-Hosted | [Self-Hosted Certificate Management](#self-hosted-certificate-management) |
| Advanced/BYOC + CMEK | [CMEK Key Management](#cmek-key-management) |
| Advanced/BYOC + TLS question | [Cloud TLS (Managed)](#cloud-tls-managed) |
| Standard | [Fully Managed Encryption](#fully-managed-encryption) |
| Basic | [Fully Managed Encryption](#fully-managed-encryption) |

---

## Self-Hosted Certificate Management

**Applies when:** Tier = Self-Hosted

### Monitor Certificate Expiry

No production-safe SQL view exposes certificate expiration. Use one of:

```bash
# Inspect certs locally on each node
cockroach cert list --certs-dir=<certs-dir>

# Or read a specific cert file
openssl x509 -in <certs-dir>/node.crt -noout -enddate

# Or scrape the per-node Prometheus endpoint (UNIX seconds for ca, node, client_ca, ui_ca)
curl -ks https://<node>:8080/_status/vars | grep '^security_certificate_expiration_'
```

Alert thresholds: CA < 1 year = plan rotation. Node < 90 days = schedule rotation. Node < 30 days = rotate immediately.

### Rotate Node Certificates (Same CA)

```bash
cockroach cert create-node <hostname> <ip> <lb-hostname> <lb-ip> localhost 127.0.0.1 \
  --certs-dir=<certs-dir> --ca-key=<ca-key-path> --overwrite
```

Deploy to node, set `chmod 0600` on key file. CockroachDB auto-detects new certs — no restart required.

See [rotation-procedures reference](references/rotation-procedures.md) for detailed steps and verification.

### Rotate CA Certificate

CA rotation requires a combined certificate (new + old) for seamless trust transition:

1. Generate new CA key and certificate
2. Create combined CA file: `cat new-ca.crt old-ca.crt > ca.crt`
3. Deploy combined CA to all nodes
4. Re-issue node and client certificates signed by the new CA
5. After all entities use new-CA-signed certs, remove old CA from combined file

See [rotation-procedures reference](references/rotation-procedures.md) for the full CA rotation procedure.

### Kubernetes Certificate Management

- **CockroachDB Operator:** Self-signer rotates automatically. Configure via `tls.certs.selfSigner.rotateCerts`.
- **cert-manager:** Auto-renews. Pods may need restart to pick up new certs.

See [kubernetes-certs reference](references/kubernetes-certs.md) for detailed Kubernetes procedures.

---

## CMEK Key Management

**Applies when:** Tier = Advanced or BYOC, CMEK enabled

### What Is CMEK

Customer-Managed Encryption Keys wrap CockroachDB's data-at-rest encryption with a key stored in your cloud provider's KMS. CockroachDB Cloud never has access to the CMEK itself.

CMEK requires an Advanced cluster with advanced security features enabled at cluster creation. It cannot be enabled retroactively. **CMEK is not available on Standard or Basic.**

### Check CMEK Status

```bash
curl -s -H "Authorization: Bearer $COCKROACH_API_KEY" \
  "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>/cmek" | jq '.'
```

Or: Cloud Console → Cluster → Security → Encryption.

### Rotate CMEK Key

Rotate the key in your cloud provider's KMS. CockroachDB Cloud automatically uses the new key version. No cluster downtime.

See [cmek-procedures reference](references/cmek-procedures.md) for provider-specific KMS rotation commands (AWS KMS, GCP Cloud KMS, Azure Key Vault) and IAM audit procedures.

### Emergency: Revoke CMEK Key

Revoking the CMEK key makes cluster data **permanently inaccessible** unless the key is restored within your KMS provider's grace period.

**Only use as an emergency kill switch.** This may be irreversible.

---

## Cloud TLS (Managed)

**Applies when:** Tier = Advanced or BYOC, question is about TLS (not CMEK)

TLS certificates are fully managed by Cockroach Labs on Advanced and BYOC:
- Provisioning, rotation, and renewal are automatic
- No customer action needed
- Certificate health is monitored by CRL

**Client certificates:** You manage your own client-side certificates for application connections. These are standard PostgreSQL client certificates.

---

## Fully Managed Encryption

**Applies when:** Tier = Standard or Basic

TLS certificates and data-at-rest encryption are fully managed by Cockroach Labs.
- No certificate visibility or rotation responsibility
- Encryption in transit is always enabled
- Encryption at rest is always enabled
- CMEK is not available on these tiers

**If CMEK is required:** Upgrade to Advanced.

---

## Safety Considerations

**Read-only monitoring queries are safe on all tiers.**

**Self-Hosted certificate operations:**
- Always backup existing certificates before rotation
- Use combined CA approach — never abruptly replace the CA
- Verify SAN entries include ALL hostnames, IPs, and load balancer addresses
- CA key must be stored separately from node certificates
- File permissions: key files must be mode 0600, owned by cockroach process user

**CMEK operations (Advanced/BYOC):**
- CMEK key revocation renders data permanently inaccessible
- Verify IAM permissions before and after KMS key rotation
- Test CMEK rotation in a staging cluster first

See [safety-guide reference](references/safety-guide.md) for detailed risk matrix.

## Troubleshooting

| Issue | Tier | Fix |
|-------|------|-----|
| Cert metric NULL | SH | Verify cluster is in secure mode |
| New cert not detected | SH | Check file permissions (0600, correct owner) |
| "unknown authority" error | SH | Deploy combined CA (new + old) |
| Connection failures after rotation | SH | Check SAN entries cover all hostnames/IPs |
| CMEK access denied | ADV/BYOC | Verify KMS key policy and IAM permissions |
| Cannot enable CMEK | ADV/BYOC | CMEK must be enabled at cluster creation |

## References

*