release-it

What this skill does

# Release It! Framework

Framework for designing, deploying, and operating production-ready software systems. Based on a fundamental truth: the software that passes QA is not the software that survives production. Production is a hostile environment -- and your system must be built to expect and handle failure at every level.

## Core Principle

**Every system will eventually be pushed beyond its design limits.** The question is not whether failures will happen, but whether your system degrades gracefully or collapses catastrophically. Production-ready software is not just correct -- it is resilient, observable, and designed to operate through partial failures without human intervention.

## Scoring

**Goal: 10/10.** When reviewing or creating production systems, rate them 0-10 based on adherence to the principles below. A 10/10 means full alignment with all guidelines; lower scores indicate gaps to address. Always provide the current score and specific improvements needed to reach 10/10.

## The Release It! Framework

Six areas that determine whether software survives contact with production:

### 1. Stability Anti-Patterns

**Core concept:** Failures propagate through integration points, cascading across system boundaries. The most dangerous patterns are not bugs in your code -- they are emergent behaviors that arise when systems interact under stress.

**Why it works:** Recognizing anti-patterns lets you identify and eliminate the cracks before production traffic finds them. Every production outage traces back to one or more of these patterns. They are predictable, recurring, and preventable.

**Key insights:**
- Integration points are the number-one killer of production systems -- every socket, HTTP call, or queue is a risk
- Cascading failures spread when one system's failure causes its callers to fail, which causes their callers to fail
- Slow responses are worse than no response -- they tie up threads, exhaust pools, and propagate delays across the entire call chain
- Unbounded result sets turn a harmless query into an out-of-memory crash when data grows beyond test assumptions
- Users generate load patterns that no test suite can predict -- bots, retry storms, and flash crowds
- Self-denial attacks occur when your own marketing, coupons, or viral features overwhelm your infrastructure
- Blocked threads are the silent killer -- deadlocks and resource contention show no errors until everything stops

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **HTTP calls** | Assume every remote call can fail, hang, or return garbage | Wrap all external calls with timeout + circuit breaker |
| **Database queries** | Enforce result set limits on every query | Add `LIMIT` clause; paginate all list endpoints |
| **Thread pools** | Isolate pools per dependency to prevent cross-contamination | Separate thread pool for payment gateway vs. search |
| **Load testing** | Simulate realistic traffic including spikes and abuse patterns | Use production traffic replays, not synthetic happy-path scripts |
| **Marketing events** | Coordinate launches with capacity planning | Pre-scale before Black Friday; add queue for coupon redemption |

See: [references/anti-patterns.md](references/anti-patterns.md) for detailed analysis of each anti-pattern with failure scenarios and detection strategies.

### 2. Stability Patterns

**Core concept:** Counter each anti-pattern with a stability pattern. Circuit breakers stop cascading failures. Bulkheads isolate blast radius. Timeouts reclaim stuck resources. Together they create a system that bends under load but does not break.

**Why it works:** These patterns work because they accept failure as inevitable and design the system's response to failure, rather than trying to prevent all failures. A circuit breaker that trips is the system working correctly -- it is protecting itself from a downstream failure.

**Key insights:**
- Circuit Breaker: three states (closed, open, half-open) -- trips after threshold failures, periodically tests recovery
- Bulkheads: partition resources so one failing component cannot drain the entire system
- Timeouts: every outbound call needs both a connect timeout and a read timeout -- and timeouts must propagate up the call chain
- Retry with backoff: exponential backoff + jitter prevents thundering herd on recovery
- Fail Fast: if you know a request will fail, reject it immediately -- do not waste resources attempting it
- Steady State: systems accumulate cruft (logs, sessions, temp files) -- design for automatic cleanup
- Let It Crash: sometimes the safest recovery is to restart the process cleanly rather than limping along in an unknown state
- Handshaking: let the server tell the client whether it can accept work before the client sends it

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Service calls** | Circuit Breaker with threshold and recovery timeout | Open after 5 failures in 60s; half-open after 30s |
| **Resource isolation** | Bulkhead with dedicated pools per dependency | Separate connection pools for critical vs. non-critical services |
| **Network calls** | Timeout with propagation | Connect: 1s, read: 5s; propagate deadline to downstream calls |
| **Retries** | Exponential backoff + jitter + retry budget | Base 100ms, max 3 retries, 20% retry budget across fleet |
| **Data cleanup** | Steady State with automated purging | Delete sessions older than 24h; rotate logs at 500MB |

See: [references/stability-patterns.md](references/stability-patterns.md) for implementation details, state machines, threshold tuning, and pattern combinations.

### 3. Capacity and Availability

**Core concept:** Capacity is not a single number -- it is a multi-dimensional function of CPU, memory, network, disk I/O, connection pools, and thread counts. Capacity planning means understanding which resource becomes the bottleneck first and at what load level.

**Why it works:** Systems that are not capacity-tested fail in production at the worst possible moment -- during peak load. Understanding your system's actual limits (not theoretical limits) lets you set realistic SLAs and plan scaling before users hit the wall.

**Key insights:**
- Performance testing taxonomy: load test (expected traffic), stress test (beyond limits), soak test (sustained load over time), spike test (sudden bursts)
- The Universal Scalability Law: throughput does not scale linearly -- contention and coherence costs cause diminishing returns
- Connection pools are finite and precious -- a pool exhaustion looks identical to a database outage from the application's perspective
- Thread pools must be sized based on measured throughput, not guesses -- too few starve the system, too many cause context-switching overhead
- Myths: "The cloud is infinitely scalable" -- auto-scaling has lag time, cold-start costs, and hard limits
- Resource pools need health checks, eviction policies, and maximum lifetime limits

**Code applications:**

| Context | Pattern | Example |
|---------|---------|---------|
| **Load testing** | Ramp to expected peak, then 2x, observe degradation curve | Gradually increase RPS until latency exceeds SLO |
| **Connection pools** | Size based on measured concurrency, not defaults | Measure active connections under load; set pool to P99 + 20% headroom |
| **Auto-scaling** | Define scaling triggers with appropriate cooldown | Scale on CPU > 70% sustained 3 min; cooldown 5 min |
| **Soak testing** | Run at 80% capacity for 24-72 hours | Catch memory leaks, connection leaks, file handle exhaustion |
| **Capacity model** | Document resource bottleneck per service | "Service X is memory-bound at 2000 RPS; needs 4GB per instance" |

See: [references/capacity-planning.md](references/capacity-planning.md) for testing methodologies, resource pool management, and scalability modeling.

### 4. Deployment and Release

**Core concept:** Deployment (putting