chaos-engineering-fundamentals

What this skill does

# Chaos Engineering Fundamentals

Principles and practices for chaos engineering - proactively discovering system weaknesses through controlled experiments.

## When to Use This Skill

- Implementing chaos engineering practices
- Designing fault injection experiments
- Building confidence in system resilience
- Discovering hidden failure modes
- Validating disaster recovery

## What is Chaos Engineering?

```text
Chaos Engineering = Proactive resilience testing

Traditional testing: "Does it work when everything is right?"
Chaos engineering: "Does it work when things go wrong?"

Principle:
Build confidence in the system's ability to withstand
turbulent conditions in production.

Not about breaking things randomly.
About controlled experiments to learn.
```

## The Chaos Engineering Loop

```text
┌─────────────────────────────────────────────────────────┐
│                CHAOS ENGINEERING LOOP                    │
│                                                          │
│   ┌─────────┐     ┌─────────┐     ┌─────────┐          │
│   │ Define  │────►│ Inject  │────►│ Observe │          │
│   │ Steady  │     │ Chaos   │     │ Results │          │
│   │ State   │     │         │     │         │          │
│   └─────────┘     └─────────┘     └────┬────┘          │
│        ▲                               │                │
│        │                               │                │
│        │          ┌─────────┐          │                │
│        └──────────│ Improve │◄─────────┘                │
│                   │ System  │                           │
│                   └─────────┘                           │
└─────────────────────────────────────────────────────────┘
```

## Core Principles

### 1. Build Hypothesis Around Steady State

```text
Steady State = Normal system behavior

Define measurable indicators:
- Request success rate: 99.9%
- Latency p99: < 200ms
- Orders processed/minute: > 100
- User sessions active: > 10,000

Hypothesis format:
"When [fault condition] occurs, the system will
maintain [steady state metrics] within [acceptable bounds]"

Example:
"When one database replica fails, request success rate
will remain above 99.5% and latency below 500ms"
```

### 2. Vary Real-World Events

```text
Inject realistic failures:

Infrastructure:
- Server crash
- Network partition
- Disk full
- CPU exhaustion
- Clock skew

Application:
- Service unavailable
- Slow responses
- Corrupted data
- Certificate expiry
- Resource exhaustion

Dependencies:
- Database failure
- Cache unavailable
- Third-party API down
- Message queue backup
```

### 3. Run Experiments in Production

```text
Why production?
- Real traffic patterns
- Real infrastructure
- Real dependencies
- Real monitoring

Start safe:
1. Begin in non-production
2. Graduate to canary
3. Progress to production
4. Expand blast radius gradually

Safety nets:
- Kill switch ready
- Rollback plan
- Limited blast radius
- Monitoring in place
```

### 4. Automate Experiments to Run Continuously

```text
One-time experiments find one-time bugs.
Continuous experiments catch regressions.

Automation goals:
- Run experiments regularly
- Integrate with CI/CD
- Catch new failure modes
- Validate changes

Example schedule:
- Critical paths: Daily
- Core services: Weekly
- Full system: Monthly
```

### 5. Minimize Blast Radius

```text
Control experiment impact:

Scope limitations:
- Single instance
- Percentage of traffic
- Specific region
- Test accounts only

Duration limits:
- Seconds to minutes
- Automatic termination
- Scheduled windows

Abort conditions:
- Error rate exceeds threshold
- Customer impact detected
- Manual kill switch
```

## Experiment Design

### Experiment Structure

```text
Experiment: [Name]
Date: [When]
Team: [Who]

## Hypothesis
When [fault is injected], the system will [expected behavior]
because [reasoning].

## Steady State Metrics
- [Metric 1]: [Expected value]
- [Metric 2]: [Expected value]

## Experiment Details
Fault Type: [What we're injecting]
Target: [Where we're injecting]
Magnitude: [How severe]
Duration: [How long]

## Blast Radius
- Affected services: [List]
- Affected users: [Percentage/count]
- Region/zone: [Scope]

## Abort Conditions
- [Condition 1] → Abort
- [Condition 2] → Abort

## Rollback Plan
1. [Step 1]
2. [Step 2]

## Results
Hypothesis: [Confirmed/Falsified]
Observations: [What we saw]
Action Items: [What to fix]
```

### Common Experiment Types

```text
1. Service Failure
   └── Kill instances, return errors

2. Network Failures
   └── Latency injection, packet loss, partitions

3. Resource Exhaustion
   └── CPU stress, memory pressure, disk full

4. Dependency Failures
   └── Database down, cache miss, API timeout

5. State Corruption
   └── Clock skew, data inconsistency

6. Traffic Surge
   └── Sudden load increase
```

## Fault Injection Patterns

### Infrastructure Faults

```text
Instance termination:
- Kill random instances
- Verify auto-scaling/recovery
- Netflix Chaos Monkey style

Zone/region failure:
- Simulate full zone outage
- Test failover to other zones
- Verify data consistency

Network partition:
- Split brain scenarios
- Cross-region communication failure
- Consensus algorithm behavior
```

### Application Faults

```text
Latency injection:
- Add artificial delay
- Test timeout handling
- Verify circuit breakers

Error injection:
- Return 500 errors
- Throw exceptions
- Test error handling paths

Resource leaks:
- Memory leaks
- Connection pool exhaustion
- File handle exhaustion
```

### Dependency Faults

```text
Database failures:
- Primary failover
- Replica lag
- Connection pool exhaustion

Cache failures:
- Cache miss scenarios
- Cache cluster failure
- Stampede protection

External API failures:
- Timeout
- Rate limiting
- Malformed responses
```

## Chaos Engineering Tools

### Open Source Tools

```text
Chaos Monkey (Netflix)
- Random instance termination
- AWS focused
- Part of Simian Army

Gremlin
- Comprehensive chaos platform
- Multiple attack types
- Enterprise features

Litmus
- Kubernetes native
- ChaosHub experiment library
- GitOps friendly

Chaos Mesh
- Kubernetes native
- Various fault types
- Dashboard included

Pumba
- Docker chaos testing
- Container-level faults
- CI/CD integration
```

### Cloud Provider Tools

```text
AWS:
- Fault Injection Simulator
- Native integration

Azure:
- Chaos Studio
- Azure-native experiments

GCP:
- No native tool (use Gremlin/Litmus)
```

## Implementation Strategy

### Maturity Model

```text
Level 0: Ad-hoc
- Manual testing
- No chaos practice
- Reactive to failures

Level 1: Beginning
- First experiments
- Non-production only
- Manual execution

Level 2: Intermediate
- Regular experiments
- Production experiments
- Some automation

Level 3: Advanced
- Continuous chaos
- Automated experiments
- Broad coverage

Level 4: Expert
- Chaos as code
- Integrated in CI/CD
- GameDays regular
```

### Getting Started

```text
Week 1-2: Foundation
- Identify critical paths
- Define steady state metrics
- Set up monitoring

Week 3-4: First Experiments
- Start with known failures
- Run in non-production
- Document learnings

Month 2: Expand
- Add more experiment types
- Move to production (carefully)
- Automate basic experiments

Month 3+: Mature
- Regular GameDays
- Continuous experiments
- Integrate with CI/CD
```

## GameDays

### What is a GameDay?

```text
GameDay = Planned chaos exercise

Like a fire drill for systems:
- Scheduled in advance
- Multiple failure scenarios
- Practice incident response
- Learn and improve
```

### GameDay Structure

```text
Before:
- Define objectives
- Plan scenarios
- Notify stakeholders
- Prepare runbooks
- Set up monitoring

During:
- Run scenarios
- Observe system behavior
- Practice incident response
- Document findings

After:
- Debrief meeting
- Document learnings
- Create action items
- Plan next GameDay
```

### GameDay Scenarios

```text
Scenario categories:

1. Infrastructure
   - Region failure
   - Network partition
   - Scaling