loop-diagnosis

What this skill does

# Loop Diagnosis

Diagnose and fix stalled agent coding loops. This skill covers the diagnostic CLI, common failure modes, and the observability patterns that prevent silent stalls.

## Quick Commands

```bash
# Diagnose all active loops at once
joelclaw loop diagnose all -c

# Diagnose a specific loop
joelclaw loop diagnose <loop-id> -c

# Diagnose AND auto-fix
joelclaw loop diagnose all -c --fix

# Full JSON output (for detailed inspection)
joelclaw loop diagnose <loop-id>
```

## What Diagnosis Checks

The `diagnose` command runs 6 checks in order:

1. **Redis state** — PRD stories (pass/skip/pending), progress entries, active claims
2. **Worktree** — exists? commits? uncommitted changes? .out files?
3. **Inngest runs** — running/failed agent-loop-* functions, recent plan runs
4. **Agent processes** — any claude/codex processes still alive?
5. **Worker health** — function_count from localhost:3111/api/inngest
6. **Diagnosis** — pattern-matches the above into a root cause

## Failure Modes & Fixes

| Diagnosis | Root Cause | Auto-Fix |
|-----------|-----------|----------|
| `CHAIN_BROKEN` | Judge sent `story.passed` but plan never received it. Event lost in transit. | Re-fires `agent/loop.story.passed` → plan picks next story |
| `ORPHANED_CLAIM` | Story claimed by an event, but agent died and no Inngest run is active. | Clears claim + re-fires plan event |
| `STUCK_RUN` | Inngest run marked RUNNING but agent process is dead. Run won't complete. | Clears claims + re-fires (manual run cancellation may be needed in Inngest dashboard) |
| `WORKER_UNHEALTHY` | Worker registering fewer functions than expected. Missing imports or crash loop. | Restarts `system-bus-worker` deployment in k8s |
| `NO_PRD` | Loop has no PRD in Redis — was nuked or never created. | None — start a new loop |
| `COMPLETE` | All stories passed or skipped. Nothing to do. | None — run `joelclaw loop nuke dead` to clean up |

## When to Use (vs Other Skills)

- **loop-diagnosis** → Loop is stuck/stalled, need to figure out why and fix it
- **loop-nanny** → Loop is running, need to monitor progress and clean up after
- **agent-loop** → Need to START a new loop

## The Event Chain

Understanding the chain helps diagnose WHERE it broke:

```
agent/loop.started
  → plan (picks story, dispatches test-writer)
    → agent/loop.story.dispatched
      → test-writer (writes acceptance tests)
        → agent/loop.tests.written
          → implement (codex/claude writes code)
            → agent/loop.story.implemented
              → review (runs tests, typecheck, claude review)
                → agent/loop.story.reviewed
                  → judge (pass/fail/retry decision)
                    → agent/loop.story.passed  ←── feeds back to plan
                    → agent/loop.story.failed  ←── feeds back to plan
                    → agent/loop.story.retry   ←── feeds back to implement
```

**Most common break point**: `judge → plan`. The `agent/loop.story.passed` event fires but plan never picks it up. This happens when:
- Inngest is restarting during the event
- Worker was restarted between judge and plan
- k8s pod restart dropped the event

## Observability Patterns

### Passive: Failure Events
Every loop function should emit failure events via `onFailure` handlers (being added by harden loop). These fire `agent/loop.function.failed` which gets logged to slog.

### Active: Watchdog (Future)
A periodic Inngest function (`system/loop-watchdog`) that:
1. Scans all loops in Redis with pending stories
2. Checks if any events were emitted in the last 10 minutes
3. If not → auto-runs diagnose + fix
4. Logs to slog + daily log

### Manual: The Diagnostic Session
When an agent needs to debug loops manually, follow this sequence:

```bash
# 1. Quick overview
joelclaw loop diagnose all -c

# 2. If fix needed
joelclaw loop diagnose all -c --fix

# 3. Verify fix worked (wait ~30s for plan to fire)
joelclaw loop status <loop-id> -c

# 4. If still stuck, check worker
curl -s localhost:3111/api/inngest | python3 -c "import json,sys; print(json.load(sys.stdin)['function_count'])"

# 5. Nuclear option: full restart
joelclaw loop restart <loop-id>
```

## Making Loops More Resilient

The root cause of most stalls is **lost events in the judge→plan chain**. Solutions being implemented:

1. **onFailure handlers** — every function gets one, logs failure + emits diagnostic event
2. **Loop watchdog** — periodic check for silent stalls
3. **Debounce on content-sync** — prevents event storms that can crowd out loop events
4. **Singleton on backfill** — prevents resource contention during loops

## Cross-References

- [agent-loop skill](/Users/joel/.pi/agent/skills/agent-loop/SKILL.md) — starting loops
- [loop-nanny skill](/Users/joel/.pi/agent/skills/loop-nanny/SKILL.md) — monitoring + cleanup
- [joelclaw skill](../joelclaw/SKILL.md) — full CLI reference
- [ADR-0028](/Users/joel/Vault/docs/decisions/0028-inngest-reliability-patterns.md) — reliability patterns