feature-flags-architect

What this skill does

# Feature Flags Architect

End-to-end feature flag design, rollout, and lifecycle management. Covers flag taxonomy (release vs ops vs experiment vs permission), gradual rollout patterns with blast-radius math, kill-switch runbooks, governance (who can flip what), and the flag-debt cleanup loop that keeps a codebase from drowning in stale flags.

This skill is provider-agnostic: the patterns work whether you run LaunchDarkly, Statsig, Unleash, Flagsmith, ConfigCat, GrowthBook, OpenFeature, or a homegrown system backed by Redis/DynamoDB/Postgres.

---

## When to use this skill

| Situation | Skill applies |
|-----------|---------------|
| Designing a flag system from scratch | Yes — start with the **flag taxonomy** section |
| Planning a risky release (DB migration, payment provider swap, framework upgrade) | Yes — use the **rollout ramp** + **kill switch** sections |
| Auditing a codebase that has hundreds of flags and growing | Yes — use the **flag debt cleanup** section + `scripts/flag_audit.py` |
| Setting up an A/B test or hold-out experiment | Partially — covers the flag side; pair with `engineering/experiment-design` for statistics |
| Building a kill switch for a third-party dependency outage | Yes — use the **kill switch runbook** generator |
| Defining who in the org can flip production flags | Yes — use the **governance** section |
| Building entitlements / paid-tier gating | Use this for the flag mechanics; use `business-growth/paywall-upgrade-cro` for the UX |

---

## Flag taxonomy — the four types

Every flag belongs to exactly one of these four types. The type determines the lifecycle, who can flip it, and whether it should auto-expire. Mixing types in one flag is a root cause of flag debt.

| Type | Purpose | Lifetime | Who flips | Auto-expire? |
|------|---------|----------|-----------|--------------|
| **Release** | Decouple deploy from release. Ship code dark, ramp to users. | Days to weeks | Engineer who owns the feature | Yes — remove after 100% rollout + 1 release |
| **Ops** | Kill switches, circuit breakers, throttles. Turn off risky behavior fast. | Permanent or long-lived | Oncall / SRE / platform team | No — but review quarterly |
| **Experiment** | A/B test, multi-arm bandit, hold-out group. Measure causal impact. | Weeks to months (test duration) | Product / data / growth | Yes — remove after winner is shipped |
| **Permission** | Entitle users to features based on plan, role, beta-list. | Permanent | Product / billing | No — but consolidate into entitlement system |

**Decision rule when adding a flag:** name the type. If you can't, the flag shouldn't exist yet — clarify intent first.

See [references/flag-types-and-patterns.md](references/flag-types-and-patterns.md) for code patterns per type, including server-side vs client-side trade-offs, evaluation-context schema, and naming conventions.

---

## Rollout strategy — the ramp

A flag without a rollout plan is just a config toggle. The rollout plan is the difference between "we shipped a feature" and "we shipped a feature without paging anyone."

### Standard ramp pattern

| Step | % | Hold | Watch | Exit criteria |
|------|---|------|-------|---------------|
| Dogfood | Internal users only | 1-3 days | Error rate, manual feedback | Zero blocker bugs |
| Canary | 1% of users | 24 hours | Error rate vs baseline, p99 latency, business KPI | No regression > 2 stdev |
| Early adopters | 5% | 24-48 hours | Same metrics | Same |
| Quarter | 25% | 24-48 hours | Same + cost / capacity | Same |
| Majority | 50% | 24-48 hours | Same | Same |
| Full | 100% | 1 week observation | Same | Plan flag removal |
| Cleanup | Remove flag | n/a | n/a | PR merged, control plane entry deleted |

This is the **safe default**. Move faster if blast radius is low; slower for irreversible / high-cost / hard-to-monitor changes.

### Blast radius math

Before flipping, compute:

```
blast_radius = (users_affected_per_pct × pct_enabled) × (avg_session_count × time_to_detect_minutes / 60)
```

Use `scripts/rollout_simulator.py` to model this for your traffic profile. Output: estimated affected users at each ramp step, ETA to detect a regression, and a recommended ramp schedule.

See [references/rollout-and-kill-switch-playbook.md](references/rollout-and-kill-switch-playbook.md) for ramp templates by change type:

- Pure UI / cosmetic — `aggressive` ramp (1% → 100% in 24h)
- New feature, additive — `standard` ramp (1 week)
- Backend migration with dual-write — `cautious` ramp (2-3 weeks)
- Schema change, irreversible writes — `conservative` ramp (4-6 weeks with cohort isolation)
- Third-party dependency swap — `phased` ramp by user-segment, never by random %

---

## Kill switches — the ops flag pattern

Kill switches are the cheapest insurance you can buy. Every external dependency, every risky code path, every feature that could go wrong should have one.

### When to add a kill switch

- New third-party SDK / API integration (e.g., new payment gateway)
- New caching layer (Redis, CDN, edge compute)
- Any code path that could explode the database (background job, sync write to slow downstream)
- AI / ML inference calls (LLM, embeddings, recommendations) — high cost, variable latency
- Any auth flow change

### Kill switch design

A good kill switch:

1. **Defaults to ON** (feature enabled). Flipping the switch turns the feature **off**, falling back to the previous behavior.
2. **Fails open or closed deliberately.** Document which, and why. Most safety kill switches fail open (allow traffic through, but skip the new feature). Some (auth, billing) must fail closed.
3. **Has a runbook attached.** Anyone on oncall can flip it without paging the owner.
4. **Is independent of the release flag.** A release flag controls rollout; a kill switch controls outage response. Don't reuse one for both.
5. **Has alerting wired in.** When flipped, the on-call rotation is paged so the team knows something failed.

Use `scripts/kill_switch_runbook.py` to generate a runbook template for a kill switch (flag name, expected behavior on/off, who to page, rollback steps, post-incident actions).

---

## Flag governance — who can flip what

The governance model is independent of the tool. Map these roles into whichever flag system you use.

| Role | Can flip | Cannot flip | Requires approval |
|------|----------|-------------|-------------------|
| Engineer (feature owner) | Release flags they own, in non-prod | Production release flags > 5%, any ops/permission flag | Yes for prod ramp steps > 5% |
| Engineer (any) | Nothing in prod | Anything in prod without approval | Yes for any prod flip |
| Oncall / SRE | Any ops flag, kill switches | Permission flags, experiments | No — but must log + post-incident review |
| Product / Growth | Experiment flags, permission flags (paid tiers) | Release, ops | Yes for experiment changes mid-flight |
| Anyone | Read-only access, evaluation history | Any flip in prod | n/a |

### Audit log requirements

Every flag flip in prod logs:
- Who flipped it (human or system, with identity)
- When (ISO 8601 timestamp, UTC)
- From what value to what value
- Justification (free-text, required for non-emergency flips)
- Related ticket / incident (required for ops flips)

Retention: 1 year minimum for release/experiment, 7 years for permission flags (legal/billing).

---

## Flag debt — the cleanup loop

Stale flags rot. They:

- Add code paths that aren't tested
- Confuse future readers ("is this still active?")
- Hide dead code from coverage tools
- Increase the chance of mis-flipping a permanent flag
- Multiply the matrix of states that QA has to verify

Every flag added without a removal plan becomes flag debt.

### The cleanup loop

Run quarterly (more often if you ship fast):

1. **Inventory** — run `scripts/flag_audit.py --path .` against your codebase. Output: every flag reference, age, type, last-flipped date, percent enabled.
2. **Classify** — for each flag: is it (a) still ramping, (b) at