system-bus

What this skill does

# System Bus Worker

The system-bus worker (`@joelclaw/system-bus`) is joelclaw's event-driven backbone — 110+ Inngest durable functions, webhook ingestion, and observability. It runs as a Hono HTTP server registered with the self-hosted Inngest instance.

## Architecture

```
packages/system-bus/
├── src/
│   ├── serve.ts                          # Hono server, Inngest registration, health endpoint
│   ├── inngest/
│   │   ├── client.ts                     # Inngest client + event type definitions
│   │   ├── middleware/                    # Gateway injection, dependency injection
│   │   └── functions/
│   │       ├── index.ts                  # Combined exports
│   │       ├── index.host.ts             # Functions for host-role worker (local Mac)
│   │       ├── index.cluster.ts          # Functions for cluster-role worker (k8s)
│   │       └── <function-name>.ts        # Individual functions
│   ├── lib/                              # Shared utilities
│   │   ├── inference.ts                  # LLM calls via pi (CANONICAL — always use this)
│   │   ├── redis.ts                      # Redis client helper
│   │   ├── typesense.ts                  # Typesense client
│   │   ├── convex-content-sync.ts        # Convex upsert for content pipeline
│   │   ├── langfuse.ts                   # Langfuse tracing
│   │   └── ...
│   ├── observability/
│   │   └── emit.ts                       # OTEL event emission
│   ├── webhooks/
│   │   ├── server.ts                     # Webhook router (mounted at /webhooks)
│   │   ├── types.ts                      # Provider interface
│   │   └── providers/                    # Per-service webhook handlers
│   │       ├── front.ts
│   │       ├── github.ts
│   │       ├── vercel.ts
│   │       ├── todoist.ts
│   │       ├── mux.ts
│   │       └── joelclaw.ts
│   └── memory/                           # Memory pipeline components
├── scripts/
│   └── sync-content-to-convex.ts         # Manual full Convex sync
└── package.json
```

## Worker Roles

Two deployment modes controlled by `WORKER_ROLE` env var:

| Role | Where | Functions |
|------|-------|-----------|
| `host` | Local Mac Mini via Talon supervisor (optional) | Agent loops, heartbeat checks, memory pipeline, content sync, video ingest, book download — anything needing local filesystem, pi CLI, or docker |
| `cluster` | k8s pod (GHCR image) | Webhooks (Front, GitHub, Vercel, Todoist, Mux), approvals, notifications, Slack backfill — stateless, network-only |

Functions are split between `index.host.ts` and `index.cluster.ts`. The combined `index.ts` exports everything for tooling/tests.

## Deployment Model

- **Source of truth**: `~/Code/joelhooks/joelclaw/packages/system-bus/`
- **Running host worker**: `worker-supervisor` process running `bun run src/serve.ts` from `~/Code/joelhooks/joelclaw/packages/system-bus/`
  - verify with `lsof -iTCP:3111 -sTCP:LISTEN -n -P` and `lsof -p <pid> | awk '$4=="cwd"{print}'`
  - legacy clone `~/Code/system-bus-worker/` may still exist, but it is not the active host worker when port 3111's cwd points at the monorepo
- **Cluster runtime**: `system-bus-worker` Deployment in the Talos/Colima k8s cluster for cluster-role workloads
- **Cluster deploy path**: `~/Code/joelhooks/joelclaw/k8s/publish-system-bus-worker.sh`

### Host function rollout reality

Host worker registration supports an explicit `INNGEST_SERVE_HOST` override in `~/.config/system-bus.env`. Set `INNGEST_SERVE_HOST=connect` to suppress SDK callback URL advertising when the self-hosted Inngest pod cannot route to the host network. Do **not** leave it pointing at a stale Tailscale or Docker-host address unless you have proven the Inngest pod can open that address from inside k8s; otherwise every host function fails with `Unable to reach SDK URL`.

After changing `packages/system-bus/src/inngest/functions/*` that run on the host worker:

1. commit + push the monorepo change to `origin`
2. confirm the live worker cwd: `pid=$(lsof -tiTCP:3111 -sTCP:LISTEN); lsof -p "$pid" | awk '$4=="cwd"{print}'`
3. if cwd is `~/Code/joelhooks/joelclaw/packages/system-bus`, kill the Bun worker PID and let `worker-supervisor` respawn it
4. if cwd is the legacy `~/Code/system-bus-worker`, sync that clone first (`git fetch origin && git reset --hard origin/main`) and then restart the process
5. verify `curl http://127.0.0.1:3111/` shows functions and `joelclaw functions` returns >0

The stale failure mode: a host worker can keep running old source for days. In that state, OTEL may show behavior that current monorepo code has already fixed. Always verify the live port-3111 process cwd and start time before debugging source that "should" already be deployed.

Queue pilot flags are evaluated inside the live worker process, not your shell. If a host-worker emitter like `discovery-capture` or `/webhooks/github` should switch to queue mode, put the flag in `~/.config/system-bus.env`, then kickstart the worker and PUT-sync `/api/inngest`. Ad-hoc shell env only affects CLI-local emitters.

Queue triage flags follow the same rule. Current bounded admission contract:
- `QUEUE_TRIAGE_MODE=off|shadow|enforce` sets the base triage mode.
- `QUEUE_TRIAGE_FAMILIES=discovery,content,subscriptions,github` (or exact event names) chooses which queue families participate at all.
- `QUEUE_TRIAGE_ENFORCE_FAMILIES=discovery,github` is the narrow Story 4 override that upgrades only `discovery/noted` and `github/workflow_run.completed` into enforce.
- Any non-eligible family is clamped back to `shadow` even if someone sets global `QUEUE_TRIAGE_MODE=enforce`.
- Handler routing always stays registry-derived; triage may only shape bounded admission fields.

`content/updated` is the odd one out: its ingress comes from the launchd watcher `com.joel.content-sync-watcher`, not from a worker-local function. The canonical watcher source now belongs in `infra/launchd/com.joel.content-sync-watcher.plist` plus `scripts/content-sync-watcher.sh`, and the script reads `~/.config/system-bus.env` on each trigger so `QUEUE_PILOTS=content` can switch between `joelclaw queue emit` and legacy `joelclaw send` without hand-editing the live plist.

For Story 5 soak work, start from `joelclaw jobs status` for the first operator glance, then drop into `joelclaw queue stats` before spelunking raw OTEL or Redis. `jobs status` is the transitional runtime surface that rolls queue / Restate / Dkron / Inngest into one JSON snapshot without forcing the operator to jump between commands just to learn whether the substrate is healthy enough to take work. `queue stats` remains the queue-specific summary for Restate drainer health and queue triage behavior: it rolls up recent `queue.dispatch.started|completed|failed` telemetry plus the `queue.triage.*` lifecycle into live depth, terminal success/failure counts, `waitTimeMs` percentiles, dispatch-duration percentiles, fallback reasons, disagreement counts, applied-vs-suggested deltas, route mismatches, family rollups, and recent mismatch/fallback samples. Use `joelclaw queue stats --since <iso|ms>` when you need to anchor the sample to a known-clean point such as a supervised `queue.drainer.started` after rollout. Honest gotcha from the live Story 5 cleanup follow-through: global depth can lie because of unrelated historical backlog, so judge the supervised sample first with the anchored triage/dispatch window plus `joelclaw queue inspect <stream-id>` / `joelclaw queue list --limit <n>` on the fresh sample IDs. If old residue survives a supervised `com.joel.restate-worker` restart, clear it with a bounded `@joelclaw/queue ack()` pass only after confirming zero pending leases and an age filter on the orphaned stream IDs. If that command is broken or misleading, fix it before widening queue cutovers.

For ADR-0217 Phase 3 Story 2-4, the operator surfaces are `joelclaw queue observe`, `joelclaw queue pause`, `joelclaw queue resume`, and `joelclaw queue control status`. `queue observe` still answers “wh