review-ai-issues

What this skill does

# Review AI-Generated Bug Reports

Walk through a set of plausibly AI-generated GitHub issues with the user, deciding for each one whether to fix it or close as not-a-bug. These reports are often plausible-sounding but wrong, pedantic, or contain regressions that reintroduce old bugs. **Be skeptical** — do not assume the report is correct just because it's detailed.

## Workflow per issue

1. **Read the issue** with `gh issue view <number>`.
2. **Read the actual code** the report cites. Do not take code snippets in the report at face value — they may be out of date or abridged.
3. **Verify the core claim.** Trace through the scenario described. Can the bug actually happen? Under what conditions? What's the realistic impact?
4. **Check git history if relevant.** If the report blames a commit, use `git show <hash>` to understand the context. Often a "bug" is a known trade-off.
5. **Summarize your assessment to the user** in a few sentences — state whether it's legit, how concerned we should be, and any alternative approaches to the proposed fix. Ask whether to fix or close.
6. **If fixing:**
   - Prefer fixing at the root cause, not just patching symptoms. If a helper function has HTTP-response semantics baked into it that's causing trouble, consider changing the helper's contract.
   - **Write a test that reproduces the bug** where reasonable. Verify the test actually fails without the fix by running `git stash push <fix files>` before running the test, then `git stash pop` after.
   - Commit with a message that references the issue (`Fixes #NNN`) and explains the *why*, not just the *what*.
7. **If closing:** close with `gh issue close <number> --comment "..."` — see template below.

## Patterns to watch for

- **"500 replaced with a different error"** — if the proposed fix just swaps an unhandled exception for a handled error page with the same user impact, it's usually not worth the complexity. Ask: does the user actually experience anything different?
- **Theoretical race conditions** — TOCTOU bugs that require two users to do conflicting things at the exact same millisecond are usually not worth `select_for_update` complexity.
- **"Security" reports where the threat doesn't apply** — e.g., "any user can POST to this endpoint" where the endpoint requires a UUID that can only come from the target's email.
- **Intentional fail-hard behavior** — e.g., refusing to delete a team if Stripe cancellation fails. Silently swallowing errors here would be worse (orphaned subscriptions). Don't "fix" these.
- **Defense-in-depth gaps** — sometimes worth fixing if cheap (e.g., adding `escape()`), but flag to the user that it's not an active vulnerability.
- **Regressions from recent fixes** — after one round of fixes, the next round often contains regressions we caused. Read the referenced commit hashes carefully and check whether our prior change broke something.

## Common causes of false positives

- **Contrived preconditions** — the bug is real but requires an absurd sequence of events (e.g., Stripe price deleted mid-checkout).
- **Outdated report** — the code has moved on since the issue was filed.
- **Misread behavior** — the report claims the function does X but it actually does Y.
- **Missing context** — e.g., claiming an endpoint is unauthenticated when it's behind a decorator added later.

## Committing

- Work on a fresh branch (`ai-audit-fixes-N` or similar) so multiple fixes stack cleanly.
- Commit each fix separately with a clear message. Reference the issue.
- Include the standard `Co-Authored-By:` trailer.
- Run the project's test command (e.g. `make test`) before committing. Fix any failures.
- **Do not push until the user confirms** — they often want to review the stack before it hits the remote.

## Closing issues

Always attribute generated comments to Claude to keep the record clear. Template:

```
*Comment generated by Claude, after discussing with the repo owner.*

<one-paragraph summary of why we're closing / what was fixed>

<if fixed: "Fixed in <commit-sha>.">
<if not-a-bug: "Closing as not worth fixing." + brief rationale>
```

## Opening the PR

Once all the issues in a round are done:
- `gh pr create` with a summary listing each issue handled and whether it was fixed or closed.
- For fixes, note what tests were added to prevent regression.
- For closes, briefly note why each wasn't worth fixing.

## Tone

- Be honest when something surprises you, including when you caused a regression.
- Push back on the report when the evidence doesn't support it. Don't apply fixes just because the report looks authoritative.
- When in doubt, discuss with the user — they have context you don't (threat model, business priorities, whether a feature is actually used).