k8s

What this skill does

# k8s Cluster Operations — joelclaw on Talos

## Architecture

```
Mac Mini (localhost ports)
  └─ Colima/Lima port forwarding (grpc for host-published service ports; avoid separate persistent autossh tunnels)
      └─ Colima VM (8 CPU, 24 GiB, 100 GiB, VZ framework, aarch64)
          └─ Docker 29.x + buildx (joelclaw-builder, docker-container driver)
              └─ Talos v1.12.4 container (joelclaw-controlplane-1, 18 GiB cap)
                  └─ k8s v1.35.0 (single node, Flannel CNI)
                      └─ joelclaw namespace (privileged PSA)
```

**⚠️ Talos has NO shell.** No bash, no /bin/sh, nothing. You cannot `docker exec` into the Talos container. Use `talosctl` for node operations and the Colima VM (`ssh lima-colima`) for host-level operations like `modprobe`.

### Colima Stability Rules (2026-03-17 incident)

| Setting | Value | Reason |
|---------|-------|--------|
| CPU | 8 | Match k8s workload requests (~2.8 CPU, 72%) |
| Memory | 24 GiB | Current post-reboot profile; 16 GiB left too little headroom once the Talos container cap is raised. Re-evaluate if macOS memory pressure returns. |
| nestedVirtualization | **OFF by default** | Crashes VM under load (image builds, heavy scheduling). Toggle ON only for Firecracker testing |
| vmType | vz | Required for Apple Silicon |
| mountType | virtiofs | Fastest option with VZ |

**`nestedVirtualization: true` is unstable on M4 Pro under load.** It causes the Colima VM to silently crash during Docker builds/pushes. Each crash:
- Kills the Talos container mid-operation
- Corrupts Redis AOF (if caught mid-write) → crash-loop on restart
- Breaks Lima socket forwarding → `docker` CLI on macOS disconnects
- Creates stale k8s pods that re-pull images → amplifies pressure

**Recovery from Colima crash-loop:**
1. `colima stop && colima start` — basic restart
2. If Redis crash-loops: `redis-check-aof --fix` (see Redis AOF Recovery below)
3. If Restate has stuck invocations: purge PVC or kill via admin API
4. If native Docker socket dead: use SSH tunnel `ssh -L /tmp/docker.sock:/var/run/docker.sock`

**Docker image builds** should use the buildx container builder (`docker buildx build --builder joelclaw-builder`) to isolate build IO from k8s workloads.

### Redis AOF Recovery

If Redis crash-loops after a VM restart with `Bad file format reading the append only file`:
```bash
# 1. Scale down Redis (or use a temp pod if StatefulSet can't mount PVC concurrently)
kubectl -n joelclaw apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: redis-fix
  namespace: joelclaw
spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  containers:
    - name: fix
      image: redis:7-alpine
      command: ["sh", "-c", "cd /data/appendonlydir && echo y | redis-check-aof --fix *.incr.aof && redis-check-aof *.incr.aof"]
      volumeMounts:
        - name: data
          mountPath: /data
  restartPolicy: Never
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-redis-0
EOF
# 2. Wait, check logs, then clean up
kubectl -n joelclaw logs redis-fix
kubectl -n joelclaw delete pod redis-fix --force
# 3. Restart Redis
kubectl -n joelclaw delete pod redis-0
```

For port mappings, recovery procedures, and cluster recreation steps, read [references/operations.md](references/operations.md).

### Reboot-heal persistence rule (2026-04-15 incident)

`infra/k8s-reboot-heal.sh` runs under launchd as a fresh process every interval. Any recovery marker that only lives in shell memory dies at the end of that tick.

That means flannel/event-healing state must be persisted on disk. Canonical path:

```bash
~/.local/state/k8s-reboot-heal.env
```

Persist at least:
- `COLIMA_START_EPOCH`
- `RECOVERY_START_EPOCH`
- `LAST_FLANNEL_RESTART_EPOCH`
- `COLIMA_UNHEALTHY_STREAK`
- `LAST_COLIMA_UNHEALTHY_EPOCH`
- `LAST_COLIMA_FORCE_CYCLE_EPOCH`
- `LAST_COLIMA_FAILED_RECOVERY_EPOCH`

Why this matters: kubelet `FailedCreatePodSandBox` events mentioning missing `subnet.env` can stay recent for minutes after the first repair. If the healer forgets that it already restarted flannel, the next launchd tick can bounce flannel again and knock healthy services like Typesense back into 503 warmup for no good reason. The extra failed-recovery marker also stops the system from counting a one-minute green flash as success and then force-cycling Colima again when the control path collapses.

### Kubeconfig / Talos Endpoint Contract (2026-05-30 incident)

Current operator access uses Colima/Lima's directly published TCP ports:

- Kubernetes API: `https://127.0.0.1:6443`
- Talos API: `127.0.0.1:50000`

Do **not** rewrite kubeconfig/Talos back to the older manual tunnel ports (`16443` / `15000`) unless explicitly testing `KUBE_OPERATOR_MODE=ssh`. After the 2026-05-30 reboot, `grpc` publishing kept service ports open but could leave Colima SSH and the Docker socket dead after load; a live `ssh` port-forwarder test did the opposite and dropped the service ports. The current availability-first default remains Colima `portForwarder: grpc`, but completion audits must verify both service ports and Docker/limactl instead of trusting either mode by vibes.

**Fix/verify**:
```bash
talosctl config endpoint 127.0.0.1:50000
talosctl config node 10.5.0.2
kubectl config set-cluster joelclaw --server=https://127.0.0.1:6443 --insecure-skip-tls-verify=true
kubectl config use-context admin@joelclaw
kubectl get nodes --request-timeout=10s
talosctl -e 127.0.0.1:50000 -n 10.5.0.2 version --client=false
```

`infra/kube-operator-access.sh` now defaults to `KUBE_OPERATOR_MODE=direct` and runs as a boring launchd monitor that keeps configs pointed at `6443` / `50000`. It still has an `ssh` mode for deliberate fallback testing.

### Durable recovery rule (ADR-0244)

A Colima restart is **not** recovery.

After any `colima start` / force-cycle, the system only counts recovery as real if a post-restart stability window stays healthy across repeated passes for:
- Colima SSH
- Docker socket
- Kubernetes API
- Typesense localhost health
- Inngest localhost health

If those regress during the verification window, classify the event as a **failed recovery**, capture proof artifacts, and stop repeated force-cycles for the configured hold period. The point is durability, not healer theatre.

## Quick Health Check

```bash
kubectl get pods -n joelclaw                          # all pods
curl -s localhost:3111/api/inngest                     # system-bus-worker → 200
curl -s localhost:7880/                                # LiveKit → "OK"
curl -s localhost:8108/health                          # Typesense → {"ok":true}
curl -s localhost:8288/health                          # Inngest → {"status":200}
curl -s localhost:9070/deployments                     # Restate admin → deployments list
curl -s localhost:9627/xrpc/_health                    # PDS → {"version":"..."}
kubectl exec -n joelclaw redis-0 -- redis-cli ping     # → PONG
joelclaw restate cron status                           # Dkron scheduler → healthy via temporary CLI tunnel
```

## Services

| Service | Type | Pod | Ports (Mac→NodePort) | Helm? |
|---------|------|-----|---------------------|-------|
| Redis | StatefulSet | redis-0 | 6379→6379 | No |
| Typesense | StatefulSet | typesense-0 | 8108→8108 | No |
| Inngest | StatefulSet | inngest-0 | 8288→8288, 8289→8289 | No |
| Restate | StatefulSet | restate-0 | 8080→8080, 9070→9070, 9071→9071 | No |
| system-bus-worker | Deployment | system-bus-worker-* | 3111→3111 | No |
| restate-worker | Deployment | restate-worker-* | in-cluster only (`restate-worker:9080`) | No |
| docs-api | Deployment | docs-api-* | 3838→3838 | No |
| LiveKit | Deployment | livekit-server-* | 7880→7880, 7881→7881 | Yes (livekit/livekit-server 1.9.0) |
| PDS | Deployment | bluesky-pds-* | 9627→**3000** | Yes (nerkho/bluesky-pds 0.4.2) |
| MinIO | StatefulSet | minio-0 | 30900→30900, 30901→30901 | No |
| Dkron | StatefulSet |