qa-changes

What this skill does

# QA Changes

Validate pull request changes by actually running the code — not just reading it. The goal is to verify that new behavior works as the PR claims, existing behavior is not broken, and the repository remains healthy after the change.

The bar is high: test the way a thorough human QA engineer would. If the PR changes a web UI, spin up the server and verify it in a real browser. If it changes a CLI, run the CLI with real inputs. Do not settle for "the tests pass" — actually use the software.

## Core Methodology

QA proceeds in four phases. Complete each phase in order. If a phase fails, report the failure and stop.

### Phase 1: Understand the Change

Read the PR diff, title, and description. **Identify the goal of this PR** — this is the single most important thing to understand before proceeding. A PR might fix a bug, add a feature, refactor code, improve performance, update documentation, or something else entirely. Check:

1. **The PR description "Why" / "Summary" section** — what is the author trying to accomplish?
2. **Linked issues** — if the PR references an issue, read it. But note: the PR may address the issue differently than expected, or only partially. The PR description is the real specification for what *this PR* intends to deliver.
3. **The PR title** — often summarizes the intent (e.g., "fix: X not working when Y", "feat: add Z capability", "refactor: consolidate duplicated X logic").

Then classify every changed file:

- **New feature**: User-visible behavior that did not exist before.
- **Bug fix**: Corrects existing behavior to match intended behavior.
- **Refactor**: Restructuring that should not change external behavior.
- **Configuration / CI / docs**: Non-functional changes.

For each change, identify the *entry point* — the concrete way a user would interact with it (CLI command, API endpoint, UI page, function call). This drives what to exercise in Phase 3.

Finally, form a clear hypothesis: "This PR should [achieve stated goal] by [approach taken in the diff]." Phase 3 will test that hypothesis.

### Phase 2: Set Up the Environment

Bootstrap the repository so the project builds and runs successfully.

1. **Read the repo's bootstrap instructions.** Check `AGENTS.md`, `README.md`, `Makefile`, `package.json`, `pyproject.toml`, `Cargo.toml`, or equivalent. Always prefer the project's own documented setup commands.
2. **Install dependencies.** Use the project's dependency manager (`uv sync`, `npm install`, `pip install -r requirements.txt`, `bundle install`, `cargo build`, etc.).
3. **Build the project** if a build step is required (compile, transpile, bundle).
4. **Note CI status.** Glance at the PR's CI checks and note whether they pass or fail. Do NOT re-run the test suite yourself — that is CI's job, not yours. Your job starts in Phase 3.

If setup fails, report the failure with the exact error output and stop.

### Phase 3: Exercise the Changed Behavior

This is the most important phase. **Actually use the software** the way a real user would to verify the change works as the PR claims. This is what distinguishes QA from CI (which runs tests) and code review (which reads code).

**Do NOT:**
- Run the test suite (`pytest`, `npm test`, `cargo test`, etc.) — that is CI's job.
- Analyze code by reading files and commenting on style, structure, or logic — that is code review's job.
- Run linters, formatters, type checkers, or pre-commit hooks — that is CI's job.

**DO:**
- Run the actual application, CLI, or server and interact with it as a user would.
- Make real HTTP requests, run real commands, open real browser pages.
- Always attempt real execution first. Running `--help`, `--dry-run`, or `--version` is NOT functional verification — it only proves argument parsing works. If real execution fails due to missing credentials, external services, or environment constraints, report what you tried and what could not be verified. Do not substitute `--help` output for evidence the software works.
- Reproduce bugs and verify fixes end-to-end.
- Test user-facing behavior that automated tests cannot or do not cover.

**Start by verifying the PR achieves its stated goal.** Use the hypothesis from Phase 1. For example:
- If the PR claims to "fix crash when X is empty", reproduce the crash scenario and confirm it no longer occurs.
- If the PR claims to "add support for Y", actually use Y end-to-end and confirm it works.
- If the PR claims to "add a new dashboard page", navigate to the page and verify it renders and functions correctly.
- If the PR claims to "add a new CLI flag", run the CLI with that flag and verify the output.

"Tests pass" is not a QA finding. The question is: does the software actually do what the PR says it does?

**For frontend / UI changes:**
- Start the development server.
- Use a real browser (via Playwright, browser automation tools, or the built-in browser) to navigate to the affected pages.
- Verify the visual change renders correctly. Take screenshots as evidence.
- Test user interactions (clicks, form submissions, navigation).
- Try at least one edge case (empty state, long text, missing data).

**For CLI changes:**
- Run the CLI command with realistic arguments. Capture stdout and stderr.
- Verify the output matches the PR's claimed behavior.
- Try at least one edge case (invalid input, missing flags, empty input).

**For API / backend changes:**
- Start the server.
- Make actual HTTP requests (`curl`, `httpie`, or a test client) to affected endpoints.
- Verify response status codes, response bodies, and side effects (database writes, file creation).
- Test error cases (bad input, missing auth, not found).

**For bug fixes — use a before/after comparison:**
1. **Reproduce the bug without the fix.** Check out the base branch (or revert the PR's changes) and run a concrete command or code path that triggers the reported failure. Show the exact command and its output.
2. **Interpret the baseline result.** Explain what the output means — e.g., "This confirms the bug exists: the resolver cannot find the package because the lockfile's cutoff date is too old."
3. **Apply the PR's changes.** Check out the PR branch, apply the patch, or set the environment variable — whatever the fix entails.
4. **Re-run the same verification.** Run the same command or exercise the same code path with the fix in place. Show the exact command and its output.
5. **Interpret the result.** Explain what the new output means — e.g., "The resolver now finds the package, confirming the fix works."
6. **Check for side effects.** Confirm the fix does not break related functionality.

**For library / SDK changes:**
- Write a short script that imports and calls the changed functions.
- Verify the return values and behavior match the PR's claims.
- Test edge cases the PR author may have missed.

**For refactors:**
- If the refactor touches a critical or user-facing path, manually exercise that path to confirm behavior is unchanged.
- For pure internal refactors where CI passes and no user-facing path is affected, Phase 2's CI check is sufficient.

**For configuration / CI / docs:**
- Validate syntax (YAML lint, JSON parse, markdown render).
- If it is a build change, confirm the build still succeeds.
- For doc changes, confirm the documentation renders correctly if a preview is available.

**Always show your work with a before/after narrative.** For every verification, the report must include: (a) the exact command you ran, (b) the actual output you observed, and (c) your interpretation of that output. For bug fixes and behavioral changes, demonstrate BOTH the broken/old state AND the fixed/new state so the reviewer can see the delta. Present this evidence inside collapsible `<details>` blocks — the core deliverable is the verdict and summary, not raw logs.

### Knowing When to Give Up

Some verification approaches will fail due to environment constraints, missing system dependencies, or tooling limitations. That is expected.

**Th