gateway-diagnose

What this skill does

# Gateway Diagnosis

Structured diagnostic workflow for the joelclaw gateway daemon. Runs top-down from process health to message delivery, stopping at the first failure layer.

**Default time range: 1 hour.** Override by asking "check gateway logs for the last 4 hours" or similar.

## CLI Commands (use these first)

```bash
# Automated health check — runs all layers, returns structured findings
joelclaw gateway diagnose [--hours 1] [--lines 100]

# Session context — what happened recently? Exchanges, tools, errors.
joelclaw gateway review [--hours 1] [--max 20]
```

Start with `diagnose` to find the failure layer. It now reports disabled launchd state for `com.joel.gateway` explicitly (instead of a generic process failure), and distinguishes `redis_degraded` from a truly dead gateway. Use `review` to understand what the gateway was doing when it broke. Only drop to manual log reading (below) when the CLI output isn't enough.

## Autonomous Monitor (cross-channel)

Gateway health is now checked automatically by Inngest function `check/gateway-health` on heartbeat fan-out event `gateway/health.check.requested`.

What it monitors:
- **General gateway failure**: critical `joelclaw gateway diagnose` layers (`process`, `cli-status`, `e2e-test`, `redis-state`)
- **Specific channel degradation**: OTEL severe action counts for `telegram-channel`, `discord-channel`, `imessage-channel`, `slack-channel`

What it does:
- Tracks failure streaks in Redis to suppress one-off noise
- Auto-restarts gateway on sustained general failure (cooldown-protected)
- Sends immediate Telegram alert only on sustained unresolved failures
- Emits OTEL with component `check-gateway-health`, action `gateway.health.checked`

## Artifact Locations

| Artifact | Path | What's in it |
|----------|------|-------------|
| **Daemon stdout** | `/tmp/joelclaw/gateway.log` | Startup info, event flow, responses, fallback messages |
| **Daemon stderr** | `/tmp/joelclaw/gateway.err` | Errors, stack traces, retries, fallback activations — **check this first** |
| **PID file** | `/tmp/joelclaw/gateway.pid` | Current daemon process ID |
| **Session ID** | `~/.joelclaw/gateway.session` | Current pi session ID |
| **Session transcripts** | `~/.joelclaw/sessions/gateway/*.jsonl` | Full pi session history (most recent by mtime) |
| **Gateway working dir** | `~/.joelclaw/gateway/` | Has `.pi/settings.json` for compaction config |
| **Launchd plist** | `~/Library/LaunchAgents/com.joel.gateway.plist` | Service config, env vars, log paths |
| **Start script** | `~/.joelclaw/scripts/gateway-start.sh` | Secret leasing, env setup, bun invocation |
| **Tripwire** | `/tmp/joelclaw/last-heartbeat.ts` | Last heartbeat timestamp (updated every 15 min) |
| **WS port** | `/tmp/joelclaw/gateway.ws.port` | WebSocket port for TUI attach (default 3018) |

## Diagnostic Procedure

Run these steps in order. Stop and report at the first failure.

### Layer -1: Substrate health (Colima + k8s)

```bash
colima status --json
kubectl get nodes -o wide
kubectl get pods -n joelclaw redis-0 inngest-0
```

**Failure patterns:**
- `colima is not running` or kubectl EOF/refused → gateway/Redis symptoms are secondary. Bring Colima back first.
- Node not `Ready` or core pods not `Running` → fix cluster substrate before touching gateway.

### Layer 0: Process Health

```bash
# Is launchd service disabled?
launchctl print-disabled gui/$(id -u) | rg "com\\.joel\\.gateway"

# Exact launchd service state (+ pid, last exit code)
launchctl print gui/$(id -u)/com.joel.gateway

# Is daemon process running outside launchd?
ps aux | grep "/packages/gateway/src/daemon.ts" | grep -v grep

# Optional PID file cross-check (missing PID file is non-fatal)
cat /tmp/joelclaw/gateway.pid
```

**Failure patterns:**
- `"com.joel.gateway" => disabled` → launchd service disabled (`joelclaw gateway enable` or `joelclaw gateway restart` to recover)
- launchctl service missing + no daemon process → gateway down
- launchd PID differs from PID file → stale PID file (degraded, not fatal)
- daemon process alive but launchd service missing/disabled → manual run or launchd drift

### Layer 1: CLI Status

```bash
joelclaw gateway status
```

**Check:**
- `mode` — `normal` vs `redis_degraded`
- `degradedCapabilities` — explicit list of what Redis loss is breaking
- `sessionPressure` — context %, compaction age, session age, next action, next threshold summary, thread counts, fallback state/activations/failures, pressure reasons, alert state
- `supersession` — latest-wins interruptibility state for human turns (active superseded request, last source/time/drop count, batching window, pending sources, last batch flush)
- `operatorTracing` — canonical Telegram operator action ack/dispatch/completion/timeout summary across callbacks + direct commands
- `callbackTracing` — compatibility alias for the same snapshot while older surfaces catch up
- `channels` — reusable runtime health/ownership snapshots for Telegram, Discord, iMessage, and Slack
- `channelHealth` — summarized degraded/muted channel state plus last degrade/recover event
- `activeSessions` — should have `gateway` with `alive: true`
- `pending: 0` — if >0, messages are backing up (session busy or stuck)

Interpretation:
- `mode: redis_degraded` means the daemon/session can still be usable while the Redis bridge is sick.
- Do not call that a full outage unless process/session layers are also failing.
- `joelclaw gateway diagnose` now emits a dedicated `session-pressure` layer so pressure risk is inspectable even when Redis/process layers are healthy.
- `interruptibility` tells you whether a newer human message already superseded the stale turn, and whether direct human channels are currently sitting inside the short batching window before dispatch, so a brief pause can be intentional instead of another silent failure.
- `operator-tracing` tells you whether Telegram operator callbacks and direct commands are acking, completing, failing, or timing out honestly, with route + trace id surfaces for the last completed/failed/timed-out action. Queued Telegram agent commands now stay open until the downstream gateway turn really finishes, and external callback-route consumers can now close traces via Redis trace-result handoff instead of being marked done at publish time.
- `channel-health` tells you whether a channel is intentionally owner/passive/fallback, truly connected, or quietly half-dead. It now also shows muted known issues and the last degrade/recover event. Telegram `fallback` with `leaseEnabled=false` is expected when poll leasing is disabled locally; do not call that degraded by itself — but if polling is actually down and only retrying after `getUpdates` conflicts, diagnose now marks that contract degraded instead of pretending fallback is healthy.
- `channel-healing` tells you whether the watchdog currently has a restart policy armed, what the degraded streak/cooldown is, whether the last heal attempt succeeded or blew up, and what manual repair summary/commands apply when the policy is `manual`. Muted degraded channels should now read as manual repair too — if you still see `restart` on a muted broken channel, the contract is lying.

### Layer 2: Error Log (the money log)

```bash
# Default: last 100 lines. Adjust for time range.
tail -100 /tmp/joelclaw/gateway.err
```

**Known error patterns:**

| Pattern | Meaning | Root Cause |
|---------|---------|-----------|
| `Agent is already processing` | Command queue tried to prompt while session streaming | Queue is not using follow-up behavior while streaming, or session is genuinely wedged |
| `dropped consecutive duplicate` | Inbound prompt was suppressed before model dispatch | Dedup collision (often from hashing channel preamble instead of message body) |
| `fallback activated` | Model timeout or consecutive failures triggered model swap | Primary model API down or slow |
| `Authentication failed for "anthropic"` | Prompt rejected before model stream sta