slo-sli-error-budget

What this skill does

# SLOs, SLIs, and Error Budgets

Patterns and practices for defining service level objectives, selecting meaningful indicators, and managing reliability through error budgets.

## When to Use This Skill

- Defining SLOs for services
- Selecting appropriate SLIs
- Implementing error budget policies
- Balancing reliability and velocity
- Setting up SLO-based alerting

## Core Concepts

### SLI (Service Level Indicator)

```text
SLI = Quantitative measure of service level

What to measure:
- Availability: % of successful requests
- Latency: % of requests faster than threshold
- Throughput: Requests per second
- Error rate: % of failed requests

Formula:
SLI = (good events / total events) × 100%

Example:
Availability SLI = (successful requests / total requests) × 100%
             = (99,500 / 100,000) × 100%
             = 99.5%
```

### SLO (Service Level Objective)

```text
SLO = Target value for an SLI

Format: SLI >= Target over Time Window

Examples:
- 99.9% of requests successful over 30 days
- 95% of requests complete in <200ms over 7 days
- 99.95% availability measured monthly

Components:
┌─────────────────────────────────────────────────────┐
│ SLO = SLI + Target + Time Window                    │
│                                                      │
│ "99.9% of HTTP requests return non-5xx             │
│  over a rolling 30-day window"                      │
└─────────────────────────────────────────────────────┘
```

### Error Budget

```text
Error Budget = Allowed unreliability

If SLO = 99.9% availability:
Error Budget = 100% - 99.9% = 0.1%

Over 30 days:
Total minutes = 30 × 24 × 60 = 43,200
Error budget = 43,200 × 0.001 = 43.2 minutes

Or in requests (assuming 1M requests/month):
Error budget = 1,000,000 × 0.001 = 1,000 failed requests

Budget consumption:
┌────────────────────────────────────────────────────┐
│ Error Budget Remaining: 65%                        │
│ ████████████████████░░░░░░░░░░                    │
│ Consumed: 35% (15 min of 43.2 min)                │
│ Days remaining in window: 12                       │
└────────────────────────────────────────────────────┘
```

## Selecting SLIs

### SLI Categories

```text
1. Request-based SLIs:
   - Availability (success rate)
   - Latency (response time)
   - Quality (correct responses)

2. Processing-based SLIs:
   - Throughput
   - Freshness (data staleness)
   - Coverage (% of data processed)

3. Storage-based SLIs:
   - Durability
   - Availability of data
```

### SLI Selection Framework

```text
For each user journey:

1. Identify critical interactions
   └── What does the user care about?

2. Map to measurable signals
   └── What can we measure?

3. Define good vs bad
   └── What's acceptable?

4. Validate with users/stakeholders
   └── Does this match expectations?
```

### Good SLI Characteristics

```text
✅ Good SLIs:
- Directly reflect user experience
- Measurable and observable
- Simple to understand
- Actionable when violated

❌ Bad SLIs:
- Internal metrics (CPU, memory)
- Too complex to explain
- Can't be measured reliably
- No clear good/bad threshold
```

### SLI Examples by Service Type

```text
API Service:
- Availability: % non-5xx responses
- Latency: % requests < 200ms
- Quality: % valid responses

Data Pipeline:
- Freshness: % data < 10 min old
- Coverage: % records processed
- Correctness: % matching expected

Storage Service:
- Durability: % objects not lost
- Availability: % successful reads
- Latency: % reads < 50ms

Web Application:
- Page Load: % pages < 3 seconds
- Interactivity: % interactions < 100ms
- Core Web Vitals: LCP, FID, CLS
```

## Setting SLO Targets

### Target Selection Approach

```text
1. Measure current performance:
   What's the baseline?

2. Understand user expectations:
   What do users need?

3. Consider business constraints:
   What's the cost of reliability?

4. Start conservative:
   Better to exceed than miss

5. Iterate based on data:
   Adjust as you learn
```

### SLO Target Guidelines

```text
Service Type        | Availability | Latency (p99)
--------------------|--------------|---------------
Internal APIs       | 99.5%        | 500ms
External APIs       | 99.9%        | 200ms
Payment systems     | 99.99%       | 300ms
Static content      | 99.95%       | 100ms
Batch processing    | 99%          | -

The "nines" scale:
99%    = 7.2 hours/month downtime
99.9%  = 43.8 minutes/month
99.99% = 4.38 minutes/month
```

### Time Windows

```text
Rolling vs Calendar:

Rolling (recommended):
- 30-day rolling window
- Smooth, no cliff effects
- Always relevant

Calendar:
- Monthly reset
- Aligns with business cycles
- Creates "budget reset" behavior

Window selection:
Short (7 days): More sensitive, faster feedback
Long (30 days): More stable, smoother trends
```

## Error Budget Policies

### Policy Components

```text
Error Budget Policy defines:

1. When to take action
   └── Budget thresholds (75%, 50%, 25%, 0%)

2. What actions to take
   └── Freeze features, focus on reliability

3. Who decides
   └── Team, management, escalation path

4. How to recover
   └── Steps to restore budget
```

### Example Policy

```text
Error Budget Policy for OrderService

Budget Remaining | Actions Required
-----------------|------------------------------------------
> 50%            | Normal development, deploy freely
25-50%           | Review deployments, increase testing
10-25%           | Freeze non-critical features
< 10%            | All hands on reliability, no new features
0% (exhausted)   | Postmortem required, leadership review

Escalation:
- Budget < 25%: Alert team lead
- Budget < 10%: Alert engineering manager
- Budget exhausted: Incident declared
```

### Budget Recovery

```text
When budget exhausted:

1. Stop non-critical deployments
2. Focus on stability improvements
3. Conduct thorough postmortems
4. Implement preventive measures
5. Resume normal work when budget recovers

Budget recovers through:
- Time passing (rolling window)
- Improved reliability
- SLO adjustment (if appropriate)
```

## Multi-Window SLOs

### Why Multiple Windows?

```text
Single window problems:
- Long window: Slow to detect issues
- Short window: Too sensitive to spikes

Solution: Multiple windows

Fast burn: Short window (1 hour)
- Detects major outages quickly
- High urgency alerts

Slow burn: Long window (30 days)
- Detects gradual degradation
- Lower urgency, more context
```

### Multi-Window Configuration

```text
Alert configuration:

Fast burn (page immediately):
- 2% of 30-day budget burned in 1 hour
- 5% of 30-day budget burned in 6 hours

Slow burn (ticket):
- 10% of 30-day budget burned in 3 days
- 20% of 30-day budget burned in 7 days

Calculation:
If 30-day budget = 43.2 minutes
2% in 1 hour = 0.864 minutes = 52 seconds of errors
→ Significant outage, page immediately
```

## SLO-Based Alerting

### Alert Design

```text
Traditional alerting:
- CPU > 80% → Alert
- Error rate > 1% → Alert
- Latency > 500ms → Alert
→ Often noisy, may not reflect user impact

SLO-based alerting:
- Error budget burn rate too high → Alert
→ Directly tied to user impact
→ Fewer, more meaningful alerts
```

### Burn Rate Calculation

```text
Burn rate = Rate of budget consumption

If budget should last 30 days:
Normal burn rate = 1x (consuming 3.33%/day)

Fast burn rate = 14.4x
→ Burning 48%/day → 0 in 2 days
→ PAGE: Major incident

Slow burn rate = 3x
→ Burning 10%/day → 0 in 10 days
→ TICKET: Needs attention
```

## Dashboard Design

### Key Metrics to Display

```text
SLO Dashboard Components:

1. Current SLI value
   └── "99.85% availability (target: 99.9%)"

2. Error budget remaining
   └── Bar chart with thresholds

3. Burn rate trend
   └── Line chart over time

4. Time to budget exhaustion
   └── "At current rate: 15 days"

5. Historical SLO compliance
   └── How often have we met SLO?

6. Key error contributors
   └── What's consuming budget?
```

### Visualization Example

```text
┌─────────────────────────────────────────────────────┐
│ OrderSer