validating-production-readiness

What this skill does

# Validating Production Readiness

This skill is the **single source of truth** for the production-readiness gate in the lt-stack. Every consumer applies the same eight pillars, the same severity classification, and produces the same report block — so multiple runs across time can be compared.

> **Goal:** Decide, with evidence, whether the system is safe to deploy to production. Anything Critical is a hard blocker. Anything Major must be tracked. Minor findings are advisory.

## When to Use This Skill

| Caller | Phase | Trigger |
|--------|-------|---------|
| `/lt-dev:production-ready` | Phase 4 | Hard release gate |
| `lt-dev:production-readiness-orchestrator` | Phase 4 | Owns the audit + remediation loop |
| `lt-dev:devops-reviewer` | (cross-reference) | Cites this checklist when flagging deployment-relevant gaps |
| Manual user invocation | Pre-deploy | Final sign-off before tagging a release |

## Severity Classification

| Severity | Meaning | Effect on the release |
|----------|---------|------------------------|
| **Critical** | Loss of data, security breach, or undetectable downtime is plausible | **BLOCK** the release until fixed |
| **Major** | Operability is degraded; recovery requires manual intervention | Tracked finding; release may proceed only with explicit owner sign-off |
| **Minor** | Cosmetic, missing nice-to-have, or hygienic gap with no operational impact | Advisory only |

A pillar with even a single Critical finding makes the whole release **NOT READY**.

## The Eight Pillars

Each pillar lists the concrete checks to perform, the file paths to inspect, and the typical fix when the check fails.

### Pillar 1 — Configuration & Secrets

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Env validation at boot | `projects/api/src/config.env.ts` (or equivalent) parses with class-validator / Valibot and crashes on missing required keys | Critical |
| `.env*` excluded from VCS | `.gitignore` covers `.env`, `.env.*`, **except** `.env.example` | Critical |
| `.env.example` exists & is current | All keys read by code are present with placeholder values | Major |
| Secrets are runtime-injected (not baked) | Production Dockerfile does NOT `COPY .env`; secrets come from CI vault | Critical |
| `NODE_ENV` taxonomy respected | `e2e` (local tests), `ci` (pipeline tests), `develop` (dev server), `test` (customer staging — NOT a test env), `production` (live) | Major |
| Per-env DB URI / database name | `nsc.mongoose.uri` ends with the env-specific DB (no shared "test" DB across envs) | Critical |
| Default-credentials guard | Production refuses to boot with `password === 'admin'` / `JWT_SECRET === 'change-me'` etc | Critical |

### Pillar 2 — Observability & Logging

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Structured logs (JSON) in production | `pino-http` or NestJS Logger configured with JSON transport when `NODE_ENV=production` | Major |
| No `console.log` in shipped server code | `grep -nR "console\.log" projects/api/src` returns 0 (`console.error` for boot failures is acceptable) | Major |
| Request ID / correlation ID | Middleware emits and propagates `x-request-id` to outgoing calls | Major |
| Error tracking wired up | Sentry (or equivalent) initialised on both API and App when DSN is set | Major |
| Metrics endpoint or APM | `/metrics` (Prometheus) OR APM agent (NewRelic/Datadog) instrumented | Minor |
| Frontend error boundary | Nuxt `error.vue` renders a useful message and reports to Sentry | Major |
| Log levels controlled by env | `LOG_LEVEL` env var read; defaults to `info` in prod, `debug` in dev | Minor |

### Pillar 3 — Health & Lifecycle

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Liveness endpoint | `GET /health` (or `/meta`) returns 200 without DB dependency, suitable for k8s liveness probe | Critical |
| Readiness endpoint | `GET /ready` performs DB ping; returns 503 when DB is unreachable | Major |
| Graceful shutdown | NestJS `app.enableShutdownHooks()` enabled; in-flight requests drain before process exit | Critical |
| Frontend SSR health | `GET /` returns 200 in SSR with mocked-or-real backend; no unhandled rejections in server log | Major |
| Docker `HEALTHCHECK` declared | Production Dockerfile includes `HEALTHCHECK` instruction calling `/health` | Major |
| Process isolation | Container runs as non-root user (`USER node` or similar) | Major |

### Pillar 4 — Security Hardening

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Helmet (or equivalent) on API | `app.use(helmet())` for REST mode; CSP set explicitly | Major |
| CORS allowlist | `cors.origin` is an explicit list, NOT `*` in production | Critical |
| HTTPS enforced | Reverse proxy / Nuxt config redirects http → https in production | Critical |
| Cookies secure | Better-Auth cookies have `secure: true`, `httpOnly: true`, `sameSite: 'lax'` (or stricter) in production | Critical |
| Rate limiting on auth | `@nestjs/throttler` (or equivalent) on `/auth/*`, `/forgot-password`, etc | Major |
| Restricted decorators on every controller | `@Restricted` / `@Roles` / `securityCheck()` present per backend-dev guidelines | Critical |
| `pnpm audit` clean (or audit ladder exhausted) | Per `running-check-script` skill rules | Major |
| No secrets in client bundle | `grep -nR "JWT_SECRET\|MONGO_URI\|API_KEY" projects/app/.output/` returns 0 after build | Critical |

### Pillar 5 — Data Durability

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Indexes for hot queries | All queries seen in load-test profiler (Pillar 6 / k6 phase) hit an index | Major |
| Migrations / seed scripts deterministic | `pnpm run migrate` is idempotent; replaying it on production data does not corrupt | Critical |
| Backup strategy documented | `README.md` or `docs/runbook.md` describes backup cadence + restore drill | Major |
| No destructive migrations on boot | Server does NOT run `dropDatabase` / `dropCollection` on startup in production | Critical |
| Soft-delete or audit trail where data loss matters | Models flagged as critical have `deletedAt` / version trail | Minor |
| Connection retry / backoff | Mongoose connect uses exponential backoff, not infinite tight loop | Major |

### Pillar 6 — Resilience Under Load

The k6 results from the `running-load-tests-with-k6` skill feed directly into this pillar. Don't re-run k6 here — read the report block.

| Check | How | Severity if missing |
|-------|-----|----------------------|
| k6 load scenario PASSED | `tests/load/summary-load.json` thresholds all PASS | Critical |
| k6 baseline regression < 15% | p95 delta vs `tests/load/baselines/` within bounds | Major |
| No memory leak in soak (when run) | `summary-soak.json` shows flat memory at end | Major |
| Connection pool not exhausted | Mongoose log shows < 80% of `maxPoolSize` at peak | Major |
| Timeouts everywhere | All HTTP / DB calls have explicit timeouts (no infinite waits) | Major |
| Idempotency keys on critical writes | Payment / email / external write endpoints have idempotency-key handling | Major |

### Pillar 7 — Deployment Hygiene

| Check | How | Severity if missing |
|-------|-----|----------------------|
| Dockerfile multi-stage | Builder + runtime stages; runtime image contains only built artefacts + prod deps | Major |
| Image base pinned | `FROM node:22.11.0-alpine` (concrete version), not `node:latest` | Major |
| `.dockerignore` complete | `.git`, `node_modules`, `tests/`, `.env*` excluded | Major |
| CI builds from main with tag | Tag-triggered pipeline builds the production image; never `latest`-only | Major |
| Build is reproducible | Lockfiles (`pnpm-lock.yaml`) committed; CI uses `--frozen-lockfile` | Major |
| Image scan in CI | `trivy` / `grype` / GitLab container scan stage exists | Major |
| Image size reasonable | Final image < 500 MB (alpine) / < 1 GB (debian-slim) | Minor |

### Pillar 8 — Run