testing-boss

What this skill does

# Testing Boss

A consolidated doctrine for writing tests that *reveal bugs*, not just pass — for human-authored code, AI-generated code, LLM-powered features, and the CI that gates them all.

The cardinal premise: **tests exist to expose defects, not to keep CI green.** A test that fails has done its job. A test that passes for the wrong reason is worse than no test.

This skill collapses the old `test-antipatterns` skill plus a much larger corpus on test placement, framework idioms, flaky-test discipline, AI-agent test generation, and LLM/agent evaluation into one self-contained body of practice. Examples are language-agnostic pseudo-code so the doctrine transfers to any stack.

## Iron Laws

```
1. Test the behavior, never the mock.
2. Push every test to the lowest layer that can detect the failure.
3. When a test fails, fix production first — change the test only after writing why.
4. Real systems gate the merge. Mocks isolate; they do not validate.
5. Coverage is a flashlight. Mutation score is a quality probe. Neither is a target.
6. No test-only methods, branches, or flags leak into production code.
```

These six laws subsume every named anti-pattern in this skill. When two of them disagree, the lower-numbered one wins.

## Required Reading Router

Match the task to the row. Read the listed file(s) **in full before** producing output. The inline content in this SKILL.md is a tripwire, not the contract.

| Task                                                        | MUST read                                                                  |
| ----------------------------------------------------------- | -------------------------------------------------------------------------- |
| Deciding where a new test belongs (layer, file, owner)      | `references/foundations.md`                                                |
| Writing a new test (any layer, any framework)               | `references/patterns.md`                                                   |
| Reviewing a test, smelling a problem, or fixing a brittle suite | `references/antipatterns.md`                                           |
| Generating tests via a coding agent (Claude Code, Codex, Cursor) | `references/ai-writes-tests.md` + `references/antipatterns.md`         |
| Triaging flaky tests, designing CI gates, or picking contract/property/mutation patterns | `references/ci-automation.md`              |
| Designing evals for LLM/agent systems (RAG, tool use, prompt regression) | `references/llm-eval.md`                                       |
| Looking up the original source for any claim in this skill  | `references/sources.md`                                                    |

## Reference Index

- **`references/foundations.md`** — placement doctrine (invariant + owning layer), pyramid vs trophy debate resolution, risk-based prioritization, coverage philosophy, test-boundary contracts.
- **`references/patterns.md`** — 12 cross-framework positive patterns with agnostic pseudo-code: selector hierarchy, condition-based waits, per-test isolation, table-driven, builders/factories, behavior-first assertions, boundary-only mocking.
- **`references/antipatterns.md`** — 25 anti-patterns across five families (Brittleness, Flakiness, Mock misuse, Process, AI-specific). Each entry: violation, why wrong, fix, gate question, evidence URL.
- **`references/ai-writes-tests.md`** — seven mandatory gates with verbatim prompt blocks for any agent that generates tests: invariant first, owning layer, real execution, failure→fix production, no snapshot without contract, no assertion on self-set mock, negative companion.
- **`references/ci-automation.md`** — flaky-test taxonomy, quarantine-plus-owner workflow, CI stage pyramid, contract / property / mutation / accessibility testing patterns, deterministic test architecture.
- **`references/llm-eval.md`** — eval-driven development primer, oracle ladder, LLM-as-judge biases and calibration, RAG metrics, agent trajectory vs outcome eval, benchmark pitfalls.
- **`references/sources.md`** — consolidated bibliography (all URLs grouped by axis) for citation and audit.

## Decide before the first line of test code

Most bad tests are placement failures, not assertion failures. A test in the wrong layer is brittle, slow, and duplicates work — or worse, it locks in implementation under the disguise of correctness.

Gist tripwires:

- Name the invariant in one sentence before opening any test file. If the sentence is fuzzy, the invariant is not clear enough to test.
- Place the test at the **lowest layer** that can fail when the invariant breaks. A higher-layer test is justified only when the invariant requires real integration the lower layer cannot prove.
- Reject the test entirely when (likelihood-of-bug × blast-radius) is below the threshold worth ten minutes of maintenance. Not every line deserves a test.

**STOP. Read `references/foundations.md` in full before placing a new test, splitting a test across layers, debating pyramid vs trophy, or arguing about coverage targets.** That file contains the placement decision tree, the explicit pyramid/trophy reconciliation, the test-boundary contract template, and the risk-based filter. The three tripwires above are detection cues, not the contract.

## Pattern catalog (write tests that survive refactor)

Twelve patterns recur across Playwright, Testing Library, Cypress, Jest, pytest, Go testing, and Pact. The framework is evidence; the principle is universal.

Named patterns (one-liners — full pseudo-code in the reference):

1. Query by behavior and accessible role, never by CSS selector or DOM index.
2. Selector hierarchy: role → label → text → test-id → structural. Stop at the highest rung that disambiguates.
3. Wait on observable conditions, never on wall-clock sleeps.
4. Each test is independent and order-free; setup beats teardown.
5. One behavior per test, but as many assertions as that behavior needs.
6. Test names read as specifications: `should <outcome> when <condition> given <state>`.
7. Table-driven / parameterized when only the inputs vary.
8. Build test data via factories or builders; literal blobs only for the field under test.
9. Mock at boundaries you do not control; real wiring for what you own.
10. Real systems gate the final merge; contract tests bridge unit and E2E.
11. Mutation score, not coverage percentage, measures suite strength.
12. Page Object Model is a tool, not a religion — collapse it for small suites.

**STOP. Read `references/patterns.md` in full before writing any non-trivial test, choosing a selector strategy, designing test data, or deciding what to mock.** That file contains the pseudo-code, the cross-framework evidence, and the explicit "when to break this rule" carve-out for each pattern. The twelve one-liners above are a vocabulary index, not the contract — the operational rule for each pattern lives only in the reference.

## Anti-pattern families (do not do these)

Twenty-five anti-patterns cluster into five families. The top seven (bolded below) cause the most damage in modern codebases — especially when AI agents write the tests.

**Brittleness** — tests bound to internals.
1. **Brittle/implementation-detail selectors.**
2. Testing internal structure instead of observable behavior.
3. Testing private methods directly.
4. Snapshot-as-test (a snapshot replacing real assertions).
5. Vague existence assertions (`.should('exist')`, `toBeTruthy`).
6. Action without assertion.

**Flakiness** — tests that randomize their own verdicts.
7. **Static `sleep` / fixed-timeout waits.**
8. **Test order dependency / hidden shared state.**
9. Non-deterministic inputs (real clock, RNG, locale).

**Mock misuse** — tests that test the test setup.
10. **Asserting the mock exists.** *(absorbed from the previous `test-antipatterns` skill)*
11. Mock drift (mock no longer matches real API).
12. Over-mocking child components.
13. Incomplete mocks (missing fields the system consumes downst