monitoring-guidelines

What this skill does

# Monitoring Guidelines

Apply these monitoring principles to ensure system reliability, performance visibility, and proactive issue detection.

## Core Monitoring Principles

- Monitor the four golden signals: latency, traffic, errors, and saturation
- Implement monitoring as code for reproducibility
- Design monitoring around user experience and business impact
- Use SLOs (Service Level Objectives) to guide alerting decisions
- Balance comprehensive coverage with actionable insights

## Key Metrics to Monitor

### Application Metrics
- Request rate (requests per second)
- Error rate (percentage of failed requests)
- Response time (p50, p90, p95, p99 latencies)
- Active connections and concurrent users
- Queue depths and processing times

### Infrastructure Metrics
- CPU utilization and load average
- Memory usage and available memory
- Disk I/O and available storage
- Network throughput and error rates
- Container and pod health (for Kubernetes)

### Business Metrics
- Transaction volumes and values
- User signups and conversions
- Feature usage and adoption rates
- Revenue-impacting events
- Customer satisfaction indicators

## Alerting Strategy

### Alert Design Principles
- Alert on symptoms, not causes
- Make alerts actionable with clear remediation steps
- Set appropriate severity levels (critical, warning, info)
- Avoid alert fatigue through proper threshold tuning
- Include runbook links in alert notifications

### SLO-Based Alerting
- Define SLOs for critical user journeys
- Calculate error budgets and burn rates
- Alert when error budget consumption is high
- Use multi-window, multi-burn-rate alerts
- Review and adjust SLOs quarterly

### Alert Configuration
- Set meaningful thresholds based on baseline data
- Use hysteresis to prevent flapping alerts
- Implement alert dependencies to reduce noise
- Route alerts to appropriate teams
- Configure escalation policies

## Dashboard Design

### Effective Dashboards
- Create overview dashboards for service health
- Build detailed dashboards for debugging
- Use consistent layouts and naming conventions
- Include time range selectors and drill-down capabilities
- Display SLO status prominently

### Dashboard Content
- Show current state and recent trends
- Include comparison to baseline or previous periods
- Display deployment markers for correlation
- Add annotations for significant events
- Include links to related dashboards and logs

## Monitoring Tools Integration

### Data Collection
- Use agents or sidecars for metric collection
- Implement service discovery for dynamic environments
- Configure appropriate scrape intervals
- Use push vs pull based on use case
- Ensure metric cardinality is manageable

### Data Storage and Retention
- Set retention periods based on use case
- Implement downsampling for long-term storage
- Use appropriate storage backends for scale
- Plan for disaster recovery of monitoring data
- Monitor your monitoring infrastructure

## Health Checks and Probes

- Implement liveness probes for crash detection
- Use readiness probes for traffic management
- Create deep health checks that verify dependencies
- Expose health endpoints in a standard format
- Monitor health check latency as a metric

## Incident Response

- Use monitoring data to detect incidents early
- Correlate metrics, logs, and traces during investigation
- Document findings and update monitoring post-incident
- Track MTTR (Mean Time to Recovery) metrics
- Conduct regular monitoring reviews and improvements

## Capacity Planning

- Track resource utilization trends
- Set alerts for approaching capacity limits
- Use forecasting for proactive scaling
- Document capacity requirements and headroom
- Review capacity quarterly