observability

What this skill does

You are an AWS observability specialist. Design monitoring, logging, and tracing solutions using CloudWatch and X-Ray.

## CloudWatch Metrics

### Key Concepts
- **Namespace**: Grouping for metrics (e.g., `AWS/EC2`, `AWS/Lambda`, custom)
- **Metric**: Time-ordered set of data points (e.g., `CPUUtilization`)
- **Dimension**: Key-value pair that identifies a metric (e.g., `InstanceId=i-xxx`)
- **Period**: Aggregation interval (60s, 300s, etc.)
- **Statistic**: Aggregation function (Average, Sum, Min, Max, p99, etc.)

### Critical Metrics by Service

| Service | Metric | Alarm Threshold | Notes |
|---|---|---|---|
| Lambda | Errors | > 0 for 1 min | Also alarm on Throttles and Duration p99 |
| Lambda | ConcurrentExecutions | > 80% of account limit | Prevent throttling |
| ALB | HTTPCode_Target_5XX_Count | > 0 for 5 min | Backend errors |
| ALB | TargetResponseTime p99 | > your SLA | Latency SLO |
| ALB | UnHealthyHostCount | > 0 | Failing targets |
| RDS | CPUUtilization | > 80% for 5 min | Sustained high CPU |
| RDS | FreeStorageSpace | < 20% of total | Prevent disk full |
| RDS | DatabaseConnections | > 80% of max | Connection exhaustion |
| DynamoDB | ThrottledRequests | > 0 | Capacity issues |
| SQS | ApproximateAgeOfOldestMessage | > your processing SLA | Queue backlog |
| ECS | CPUUtilization / MemoryUtilization | > 80% for 5 min | Scaling trigger |

### Custom Metrics
- Use `PutMetricData` API or the CloudWatch Agent
- Embedded Metric Format (EMF) for Lambda: log structured JSON that CloudWatch automatically extracts as metrics. Zero API calls, no cost per PutMetricData.
- High-resolution metrics (1-second) cost more — use only when sub-minute granularity matters
- Metric math: combine metrics without publishing new ones (e.g., error rate = Errors / Invocations * 100)

## CloudWatch Logs

### Log Groups and Retention
- Set retention on every log group. The default is **never expire** — this gets expensive fast.
- Recommended: 30 days for dev, 90 days for production, archive to S3 for long-term
- Use subscription filters to stream logs to Lambda, Kinesis, or OpenSearch

### Structured Logging
Always log in JSON format. This enables Logs Insights queries on fields.

```json
{"level": "ERROR", "message": "Payment failed", "orderId": "123", "errorCode": "DECLINED", "duration_ms": 45}
```

### CloudWatch Logs Insights Queries

```
# Find errors in Lambda functions
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 100

# P99 latency from structured logs
fields @timestamp, duration_ms
| stats percentile(duration_ms, 99) as p99, avg(duration_ms) as avg_ms by bin(5m)

# Top 10 most frequent errors
fields @timestamp, errorCode, @message
| filter level = "ERROR"
| stats count(*) as error_count by errorCode
| sort error_count desc
| limit 10

# Request rate over time
fields @timestamp
| stats count(*) as requests by bin(1m)
| sort @timestamp desc

# Find slow requests
fields @timestamp, @duration, @requestId
| filter @duration > 5000
| sort @duration desc
| limit 20

# Cold starts in Lambda
filter @type = "REPORT"
| fields @requestId, @duration, @initDuration
| filter ispresent(@initDuration)
| stats count(*) as cold_starts, avg(@initDuration) as avg_init by bin(1h)

# API Gateway latency breakdown
fields @timestamp
| filter @message like /API Gateway/
| stats avg(integrationLatency) as backend_ms, avg(latency) as total_ms by bin(5m)
```

## CloudWatch Alarms

### Alarm Types
- **Static threshold**: Fixed value (e.g., CPU > 80%)
- **Anomaly detection**: ML-based band. Good for metrics with patterns (traffic, latency).
- **Composite alarm**: Combine multiple alarms with AND/OR logic. Reduces noise.

### Alarm Best Practices
- Use **3 out of 5 datapoints** evaluation to avoid flapping on transient spikes
- Set `TreatMissingData` to `notBreaching` for low-traffic services (avoids false alarms when no data)
- Set `TreatMissingData` to `breaching` for critical health checks (missing data = something is down)
- Use composite alarms to create "alarm hierarchies": a top-level alarm that fires only when multiple sub-alarms are in ALARM state
- Always send alarms to SNS. Connect SNS to PagerDuty, Slack, or email.

### Anomaly Detection
- Trains on 2 weeks of data. Do not enable during a known-bad period.
- Adjust the band width (number of standard deviations). Start with 2, widen if too noisy.
- Best for: request count, latency, error rate — metrics with daily/weekly patterns.
- Not good for: binary metrics, metrics that are normally zero.

## CloudWatch Dashboards

### Dashboard Design
- One dashboard per service or domain (not one giant dashboard)
- Top row: key business metrics (request rate, error rate, latency p99)
- Second row: infrastructure health (CPU, memory, connections)
- Third row: dependencies (downstream API latency, queue depth)
- Use metric math to show rates and percentages, not raw counts
- Add text widgets to document what each section monitors and what to do when values are abnormal

### Automatic Dashboards
- CloudWatch provides automatic dashboards per service — start there before building custom
- ServiceLens provides an application-centric view combining metrics, logs, and traces

## X-Ray Tracing

### When to Use X-Ray
- Distributed applications with multiple services
- Debugging latency issues across service boundaries
- Understanding request flow and dependencies

### Instrumentation
- AWS SDK automatically instruments calls to AWS services
- Use X-Ray SDK or OpenTelemetry to instrument your application code
- Set sampling rules to control trace volume (default: 1 req/sec + 5% of additional)

### Key X-Ray Concepts
- **Trace**: End-to-end request path
- **Segment**: A single service's processing of the request
- **Subsegment**: Detailed breakdown within a segment (DB call, HTTP call)
- **Service Map**: Visual representation of your architecture based on trace data
- **Annotations**: Indexed key-value pairs for filtering traces (e.g., `customerId=123`)
- **Metadata**: Non-indexed data attached to segments

### X-Ray Best Practices
- Add annotations for business-relevant fields (user ID, order ID) so you can filter traces
- Use groups to define filter expressions for specific trace sets
- Active tracing on API Gateway and Lambda captures the full request lifecycle
- X-Ray daemon runs as a sidecar in ECS or as a DaemonSet in EKS

## Contributor Insights

- Identifies top contributors to a metric (e.g., top IPs, top API callers)
- Define rules in JSON that specify log group + fields to analyze
- Good for: identifying noisy neighbors, DDoS sources, hot partition keys in DynamoDB

## Common CLI Commands

```bash
# Query Logs Insights
aws logs start-query --log-group-name /aws/lambda/my-function \
  --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | limit 20'

# Get query results
aws logs get-query-results --query-id "query-id-here"

# Describe alarms in ALARM state
aws cloudwatch describe-alarms --state-value ALARM --query 'MetricAlarms[*].{Name:AlarmName,Metric:MetricName,State:StateValue}'

# Get metric statistics
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors \
  --start-time 2024-01-01T00:00:00Z --end-time 2024-01-01T01:00:00Z \
  --period 300 --statistics Sum --dimensions Name=FunctionName,Value=my-function

# Put custom metric
aws cloudwatch put-metric-data --namespace MyApp --metric-name RequestLatency \
  --value 42 --unit Milliseconds --dimensions Name=Environment,Value=prod

# List log groups with retention
aws logs describe-log-groups --query 'logGroups[*].{Name:logGroupName,RetentionDays:retentionInDays,StoredBytes:storedBytes}'

# Set log retention
aws logs put-retention-policy --log-group-name /aws/lambda/my-function --retention-in-days 30

# List X-Ray traces
aws xray get-trace-summaries --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s)

# Get X-Ray