qa-docs-coverage

What this skill does

# QA Docs Coverage (Jan 2026) - Discovery, Freshness, and Runbook Quality

## Modern Best Practices (January 2026)

- **Docs as QA**: Treat docs as production artifacts with owners, review cadence, and CI quality gates (links/style/contracts/freshness)
- **Contract-first**: Validate OpenAPI/AsyncAPI/JSON Schema in CI; use coverage tools (Swagger Coverage / OpenAPI Coverage) to detect gaps
- **Runbook testability**: Every runbook must be executable in staging; validate with synthetic tests and incident exercises
- **Automation + observability**: Track coverage %, freshness, and drift via CI dashboards; prevent regressions via PR checklists

This skill provides operational workflows for auditing existing codebases, identifying documentation gaps, and systematically generating missing documentation. It complements [docs-codebase](../docs-codebase/SKILL.md) by providing the **discovery and analysis** layer.

**Key Principle**: Templates exist in docs-codebase. This skill tells you **what** to document and **how to find** undocumented components.

Core references: [Diataxis](https://diataxis.fr/) (doc structure), [OpenAPI](https://spec.openapis.org/oas/latest.html) (REST), [AsyncAPI](https://www.asyncapi.com/) (events).

## When to use

- Auditing an existing repo for missing/outdated documentation
- Adding documentation quality gates (lint/link checks/contracts/freshness) to CI/CD
- Validating runbooks for incident readiness (MTTR reduction)

## When to avoid

- Writing new documentation from scratch without a component inventory (use discovery first)
- Publishing AI-generated docs without human review and command/link verification

## Quick start

Use progressive disclosure: load only the reference file you need.

1. Discover components: [references/discovery-patterns.md](references/discovery-patterns.md)
2. Measure coverage + gaps: [references/audit-workflows.md](references/audit-workflows.md) (Phase 1-2) and [assets/coverage-report-template.md](assets/coverage-report-template.md)
3. Prioritize work: [references/priority-framework.md](references/priority-framework.md)
4. Create an actionable backlog: [assets/documentation-backlog-template.md](assets/documentation-backlog-template.md) and templates in [docs-codebase](../docs-codebase/SKILL.md)
5. Prevent regression: [references/cicd-integration.md](references/cicd-integration.md) and [references/freshness-tracking.md](references/freshness-tracking.md)

Optional (recommended scripts; run from the repo being audited):

- Local link check: `python3 frameworks/shared-skills/skills/qa-docs-coverage/scripts/check_local_links.py docs/`
- Freshness report: `python3 frameworks/shared-skills/skills/qa-docs-coverage/scripts/docs_freshness_report.py --docs-root docs/`

### Docs Folder / LLM Iteration Audit (Critical Option)

Use this when any repository has a `docs/` folder with many LLM-generated research and implementation artifacts across phases/iterations.

1. Inventory docs and classify by type (`Tutorial`, `How-to`, `Reference`, `Explanation`).
2. Detect duplicate topics and define one canonical file per topic/feature.
3. Audit claim quality:
   - external claims must include source link + verification date
   - implementation claims must map to current code or decision log
4. Enforce lifecycle metadata for non-canonical docs (`status`, `integrates_into`, `owner`, `last_verified`, `delete_by`).
5. Trim aggressively at each phase boundary: integrated drafts must be deleted on schedule; track rare retention exceptions in backlog.

Minimum QA gate for docs folders:
- block merge if a canonical doc is missing for a changed feature
- block merge if `delete_by` is passed for `integrated` docs
- block merge on broken links or stale critical docs without owner
- block merge if `AGENTS.md` or `README.md` is missing, stale, or not linked to current canonical docs

---

## Large Codebase Audit (100K-1M LOC)

For large codebases, the key principle is: **LLMs don't need the entire codebase - they need the right context for the current task**.

### Phase 0: Context Extraction

Before starting an audit, extract codebase context using tools:

| Tool | Command/URL | Use Case |
|------|-------------|----------|
| **gitingest** | Replace "github.com" with "gitingest.com" | Quick full-repo dump |
| **repo2txt** | https://github.com/kirill-markin/repo2txt | Selective file extraction |
| **tree** | `tree -L 3 --dirsfirst -I 'node_modules|.git|dist'` | Structure overview |

### Hierarchical Audit Strategy

For monorepos and large projects, audit hierarchically:

```text
1. Root Level (Week 1)
   ├── AGENTS.md / CLAUDE.md exists?
   ├── README.md quality
   ├── ARCHITECTURE.md exists?
   └── docs/ directory structure

2. Module Level (Week 2-3)
   ├── Each major directory has AGENTS.md?
   ├── API documentation complete?
   └── Service boundaries documented?

3. Component Level (Week 4+)
   ├── Individual component READMEs
   ├── Code comments quality
   └── Test documentation
```

### Cross-Platform Documentation Audit

Check for multi-tool compatibility:

```text
[ ] AGENTS.md exists (cross-platform standard)
[ ] CLAUDE.md exists or symlinked to AGENTS.md
[ ] GEMINI.md symlinked (if using Gemini)
[ ] File size under 300 lines (use @references for depth)
[ ] Subdirectory docs for each major module
```

### Large Codebase Coverage Checklist

```text
LARGE CODEBASE AUDIT CHECKLIST

Context Extraction:
[ ] Generated codebase dump (gitingest/repo2txt)
[ ] Created directory structure overview
[ ] Identified major modules/services

Root Documentation:
[ ] AGENTS.md / CLAUDE.md present and <300 lines
[ ] README.md with quick start
[ ] ARCHITECTURE.md with system overview
[ ] Symlinks configured for cross-platform

Module Documentation:
[ ] Each major directory has AGENTS.md
[ ] API endpoints documented
[ ] Database schemas documented
[ ] Event/message contracts documented

Maintenance:
[ ] Documentation ownership assigned
[ ] Freshness tracking enabled
[ ] CI/CD checks configured
```

**Sources**: [Anthropic Claude Code Best Practices](https://www.anthropic.com/engineering/claude-code-best-practices), [OpenAI AGENTS.md Guide](https://developers.openai.com/codex/guides/agents-md)

---

## Core QA (Default)

### What "Docs as QA" Means

- Treat docs as production quality artifacts: they reduce MTTR, enable safe changes, and define expected behavior.
- REQUIRED doc types for reliability and debugging ergonomics:
  - "How to run locally/CI" and "how to test"
  - Operational runbooks (alerts, common failures, rollback)
  - Service contracts (OpenAPI/AsyncAPI) and schema examples
  - Known issues and limitations (with workarounds)

### Coverage Model (Risk-Based)

- Prioritize docs by impact:
  - P1: externally consumed contracts and failure behavior (OpenAPI/AsyncAPI, auth, error codes, SLOs).
  - P2: internal integration and operational workflows (events, jobs, DB schema, runbooks).
  - P3: developer reference (configs, utilities).

### Freshness Checks (Prevent Stale Docs)

- Define owners, review cadence, and a "last verified" field for critical docs.
- CI economics:
  - Block PRs only for missing/invalid P1 docs.
  - Warn for P2/P3 gaps; track via backlog.
- Run link checks and linting as fast pre-merge steps.

### Runbook Testability

- A runbook is "testable" if a new engineer can follow it and reach a measurable end state.
- Include: prerequisites, exact commands, expected outputs, rollback criteria, and escalation paths.

### Do / Avoid

**Do**:

- Keep docs close to code (same repo) and version them with changes.
- Use contracts and examples as the source of truth for integrations.

**Avoid**:

- Large ungoverned `docs/` folders with no owners and no CI gates.
- Writing runbooks that cannot be executed in a sandbox/staging environment.

---

## Quick Reference

| Audit Task | Tool/Pattern | Output | Reference |
| ---------- | ------------ | ------ | --------- |
| **Discover APIs** | `**/*Controller.cs`, `**/routes/**/*.ts` | Compon