agent-guardrails-update

What this skill does

# Agent Guardrails Update

Re-analyze session logs against installed guardrail rules to measure effectiveness and refine patterns. This is the feedback loop: install rules with `/agent-guardrails:install`, use them for a while, then run `/agent-guardrails:update` to tune.

## Workflow

### Step 1: Load Installed Rules

Read the installed guardrail script:

```bash
cat .claude/hooks/stop-guardrails.sh 2>/dev/null
```

Parse each grep block to extract:
- Rule name (from the `# {name}` comment)
- Pattern (from the `grep -qiE` argument)
- Message (from the `blocked+=` argument)

If no script exists, tell the user to run `/agent-guardrails:install` first and stop.

### Step 2: Load Canonical Rule Definitions

Read the canonical rule definitions from the plugin's `rules/` directory:

**Source of truth:** `${CLAUDE_PLUGIN_ROOT}/rules/no-*.md`

Read each rule file's YAML frontmatter to extract `name`, `pattern`, and `message` fields. The six baseline categories are: `no-guessing`, `no-stalling`, `no-preference-asking`, `no-false-completion`, `no-skipping`, `no-dismissing`.

### Step 3: Analyze Recent Sessions

Find session logs from the period since rules were installed. Default to last 7 days; accept `$ARGUMENTS` for custom ranges (e.g., "update last 30 days").

```bash
find ~/.claude/projects/ -name "*.jsonl" -mtime -7 ! -path "*/subagents/*" 2>/dev/null | head -30
```

Build and run a Python analysis script at `/tmp/agent-guardrails-update.py` that:

1. Reads each JSONL session file
2. Extracts assistant message text from `type: "assistant"` entries
3. For each installed rule, tests every assistant message against the rule's regex pattern
4. Records:
   - **Matches per rule** — how many times each rule's pattern was found in raw assistant text
   - **Match excerpts** — 200-char context around each match (up to 10 per rule)
   - **Phrase frequency** — which specific phrases trigger each rule most often
5. Also runs a full anti-pattern scan using the canonical rule definitions to find **uncovered patterns** — anti-patterns present in logs that no installed rule catches
6. Outputs structured JSON to stdout

**Script requirements:**
- Python stdlib only (json, re, os, glob, sys, collections)
- Handle malformed JSONL gracefully
- Skip subagent sessions
- Accept `--days N` argument

### Step 4: Assess Rule Effectiveness

For each installed rule, classify it into one of four states:

#### State: Working Well
- Pattern matches anti-pattern instances correctly
- No evidence of false positives (matches on legitimate content)
- **Action:** Keep as-is. Report match count.

#### State: Too Aggressive (False Positives)
Detected when the pattern matches legitimate assistant text. Common indicators:
- Rule matches inside code blocks or tool output (not assistant prose)
- Rule matches in contexts where the phrase is appropriate (e.g., "I think" in "I think we should" when giving a recommendation the user asked for)
- Very high match count relative to other rules (over-triggering)

**Action:** Suggest tightening the regex. Provide the specific false-positive excerpts and a refined pattern.

#### State: Too Lenient (False Negatives)
Detected when the full anti-pattern scan finds instances that the installed rule misses. Common indicators:
- Novel phrases that express the same anti-pattern but don't match the regex
- Variations or synonyms the pattern doesn't cover
- Multi-sentence hedging that no single-phrase regex catches

**Action:** Suggest expanding the regex with the missed phrases. Provide the specific missed excerpts and an updated pattern.

#### State: Inactive / No Data
- Rule is enabled but had zero matches in the analysis period
- Either the rule is working perfectly (preventing the behavior) or the pattern never triggers

**Action:** Note as "no matches" — cannot determine effectiveness without matches. Suggest the user check if the behavior was actually prevented or if the pattern needs testing.

### Step 5: Check for Uncovered Patterns

Compare the full anti-pattern scan results against installed rules. For each anti-pattern category:

- If a rule exists for it -> covered (assess effectiveness above)
- If no rule exists -> **uncovered gap**

Report uncovered gaps with:
- Category name
- Match count from scan
- Top phrases found
- Recommended rule (use curated rule from `/agent-guardrails:install` if available)

### Step 6: Present Update Report

Format results inline:

```
## Agent Guardrails Update Report

**Rules analyzed:** {count}
**Sessions scanned:** {count}
**Time range:** {date range}

### Rule Effectiveness

| Rule | Status | Matches | Assessment |
|------|--------|---------|------------|
| no-guessing | enabled | 23 | Working well |
| no-stalling | enabled | 5 | Too aggressive (2 false positives) |
| no-preference-asking | enabled | 0 | No data |
| no-false-completion | enabled | 8 | Working well |
| no-skipping | enabled | 12 | Too lenient (missed 4 instances) |

### Detailed Findings

#### no-stalling — Too Aggressive
**False positive excerpts:**
> "...let me first check if the file exists before writing to it..." (line 234, session abc123)
> This is legitimate — checking before writing is correct behavior, not stalling.

**Suggested pattern refinement:**
Remove "let me first check" from the pattern since it often occurs in legitimate pre-action verification.

**Current pattern:**
`(?i)(...|let me first check|...)`

**Proposed pattern:**
`(?i)(...|let me (first )?explain|...)`
(removed "let me first check" — kept "let me first explain" which is actual stalling)

---

#### no-skipping — Too Lenient
**Missed instances:**
> "...I won't bother with the edge cases since they're unlikely..." (not matched)
> "...that's close enough for now..." (not matched)

**Suggested pattern additions:**
Add: `I won'?t bother|close enough|good enough for now|that'?ll do`

**Proposed updated pattern:**
`(?i)(...existing...|I won'?t bother|close enough|good enough for now|that'?ll do)`

---

### Uncovered Gaps

| Category | Matches Found | Has Rule? |
|----------|--------------|-----------|
| Excessive apologizing | 7 | No |

**Suggested new rule:** Add a `# no-apologizing` grep block to `.claude/hooks/stop-guardrails.sh`

### Recommendations

1. **Update no-stalling** — tighten pattern to reduce false positives
2. **Update no-skipping** — expand pattern to catch missed phrases
3. **Install no-apologizing** — new pattern detected, not yet covered
4. **Keep no-guessing** — working well, no changes needed
5. **Monitor no-preference-asking** — no data yet, keep enabled
```

### Step 7: Apply Updates

After presenting the report, apply the recommended changes:

1. **For pattern refinements:** Edit `.claude/hooks/stop-guardrails.sh` with the updated grep pattern using the Edit tool
2. **For new rules:** Add a new grep block to `.claude/hooks/stop-guardrails.sh` following the existing pattern

After each edit, read the file back to verify the change was applied correctly.

Report what was changed:

```
## Changes Applied

- Updated `no-stalling` pattern in `.claude/hooks/stop-guardrails.sh` — removed false-positive trigger
- Updated `no-skipping` pattern in `.claude/hooks/stop-guardrails.sh` — added 4 new phrases
- Added `no-apologizing` rule to `.claude/hooks/stop-guardrails.sh` — new grep block

Script changes are active immediately.
```

## Edge Cases

**If all rules are working well:** Report that with match counts and congratulate the user on well-tuned rules. Suggest running update again after more sessions accumulate.

**If no session logs exist:** Tell the user to run some sessions first, then come back to update.

**If rules were recently installed (< 2 days):** Note that the sample size is small and results may not be representative. Still run the analysis but flag the limitation.

**If the user has custom rules not in the curated set:** Respect the customization. Base refinement suggestions on the user's version, not the curated original.