python-micrometer-sli-slo-monitoring

What this skill does

# Micrometer SLI/SLO Monitoring

## Table of Contents

1. [Purpose](#purpose)
2. [When to Use](#when-to-use)
3. [Quick Start](#quick-start)
4. [Instructions](#instructions)
5. [Examples](#examples)
6. [Requirements](#requirements)
7. [Anti-Patterns to Avoid](#anti-patterns-to-avoid)
8. [See Also](#see-also)

---

## Purpose

Service Level Objectives (SLOs) define reliability targets (e.g., 99.9% availability, P95 latency < 500ms). Service Level Indicators (SLIs) measure whether SLOs are met. Micrometer's histogram buckets (Service Level Objectives in Micrometer) allow you to define thresholds aligned with business SLOs, enabling automatic error budget calculations and SLO compliance tracking.

## When to Use

Use this skill when you need to:

- **Define SLOs for services** - Set reliability targets (availability, latency, throughput) based on business requirements
- **Implement SLI measurement** - Configure metrics to track whether SLOs are being met
- **Create SLO-aligned histogram buckets** - Define buckets at SLO thresholds (e.g., 500ms for latency)
- **Monitor error budgets** - Track remaining error budget before SLO violation
- **Set up SLO-based alerting** - Alert when approaching or violating SLO thresholds
- **Calculate SLO compliance** - Query metrics to determine percentage of requests meeting SLO
- **Implement multi-tier SLOs** - Different reliability targets for different endpoints (P1/P2/P3)
- **Track burn rate** - Detect when error budget is consumed too quickly

**When NOT to use:**
- Before defining business SLOs (work with product/business teams first)
- For purely technical metrics without SLO targets (use `python-micrometer-core` instead)
- When Micrometer isn't set up (use `python-micrometer-metrics-setup` first)
- For high-cardinality metrics (use `python-python-micrometer-cardinality-control` to prevent metric explosion)

---

## Quick Start

Define SLOs as histogram buckets in configuration:

```yaml
management:
  metrics:
    distribution:
      # Histogram buckets aligned with business SLOs
      slo:
        http.server.requests: 10ms,50ms,100ms,200ms,500ms,1s,2s,5s

      # Enable histogram export
      percentiles-histogram:
        http.server.requests: true
```

Then query SLO compliance in Prometheus/Stackdriver:

```promql
# Availability SLI: % requests without errors (non-5xx)
sum(rate(http_server_requests_seconds_count{status!~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
* 100

# Latency SLI: % requests faster than 500ms SLO
sum(rate(http_server_requests_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
* 100
```

## Instructions

### Step 1: Define Business SLOs

Start by identifying meaningful SLOs for your service:

**Typical SLO Categories:**

1. **Availability SLO**
   - Goal: 99.9% of requests succeed (no 5xx errors)
   - Period: 30 days rolling window
   - Error Budget: 0.1% errors = ~43 minutes downtime/month

2. **Latency SLO**
   - Goal: 95% of requests complete within 500ms
   - Measurement: p95 latency
   - SLI: percentage of requests faster than threshold

3. **Throughput SLO**
   - Goal: Handle 1000 requests/second peak
   - Measurement: request rate
   - SLI: achieved RPS vs. target

**Example SLOs for Supplier Charges API:**

```
Service: supplier-charges-api

Availability SLO:
  Target: 99.9%
  Window: 30 days
  Error Budget: 43 minutes/month
  SLI: (successful_requests / total_requests) >= 0.999

Latency SLO:
  Target: p95 < 500ms, p99 < 1s
  Window: 30 days
  SLI: percentage of requests meeting threshold >= 95%

Charge Processing SLO:
  Target: 99% of charges approved within 5 minutes
  Window: 7 days
  SLI: (charges_approved_within_5m / total_charges) >= 0.99
```

### Step 2: Configure SLO Histogram Buckets

Map SLO thresholds to histogram buckets in Micrometer:

```yaml
# application.yml
management:
  metrics:
    distribution:
      # Define buckets matching your SLOs
      slo:
        # HTTP request latencies: include SLO thresholds
        http.server.requests: |
          10ms,50ms,100ms,200ms,500ms,1s,2s,5s

        # Client request latencies
        http.client.requests: |
          100ms,500ms,1s,5s

        # Custom business metrics (milliseconds)
        charge.approval.duration: |
          1000,5000,10000,30000,60000

        # Invoice generation time
        invoice.generation.duration: |
          100,500,1000,5000,10000

      # Enable histogram export (required for Prometheus/Stackdriver)
      percentiles-histogram:
        http.server.requests: true
        http.client.requests: true
        charge.approval.duration: true
        invoice.generation.duration: true
```

**Bucket Strategy:**
- Include SLO threshold (500ms for availability)
- Add boundaries above and below for tail analysis
- Keep 5-10 buckets (too many = cardinality explosion)
- Use powers of 2 or multiples for readability

```java
// Alternative: programmatic configuration
@Configuration
public class SLOMetricsConfig {

    @Bean
    public MeterRegistryCustomizer<MeterRegistry> sloHistograms() {
        return registry -> {
            registry.config().meterFilter(
                new MeterFilter() {
                    @Override
                    public DistributionStatisticConfig configure(
                            Meter.Id id,
                            DistributionStatisticConfig config) {

                        // HTTP request SLOs
                        if (id.getName().equals("http.server.requests")) {
                            return DistributionStatisticConfig.builder()
                                .percentilesHistogram(true)
                                .serviceLevelObjectives(
                                    Duration.ofMillis(10).toNanos(),
                                    Duration.ofMillis(50).toNanos(),
                                    Duration.ofMillis(100).toNanos(),
                                    Duration.ofMillis(200).toNanos(),
                                    Duration.ofMillis(500).toNanos(),  // SLO threshold
                                    Duration.ofSeconds(1).toNanos(),
                                    Duration.ofSeconds(2).toNanos(),
                                    Duration.ofSeconds(5).toNanos()
                                )
                                .build()
                                .merge(config);
                        }

                        // Charge approval SLOs (5 minute target)
                        if (id.getName().equals("charge.approval.duration")) {
                            return DistributionStatisticConfig.builder()
                                .percentilesHistogram(true)
                                .serviceLevelObjectives(
                                    Duration.ofSeconds(10).toNanos(),
                                    Duration.ofSeconds(60).toNanos(),
                                    Duration.ofMinutes(5).toNanos(),     // SLO threshold
                                    Duration.ofMinutes(10).toNanos()
                                )
                                .build()
                                .merge(config);
                        }

                        return config;
                    }
                }
            );
        };
    }
}
```

### Step 3: Implement SLI Calculation Queries

Create monitoring queries to measure SLI compliance:

**Prometheus Queries:**

```promql
# === AVAILABILITY SLI ===
# Goal: 99.9% success rate (non-5xx responses)

# Current availability (5-minute window)
(
  sum(rate(http_server_requests_seconds_count{status!~"5.."}[5m]))
  /
  sum(rate(http_server_requests_seconds_count[5m]))
) * 100

# 30-day rolling availability (error budget)
(
  sum(rate(http_server_requests_seconds_count{status!~"5.."}[30d]))
  /
  sum(rate(http_server_requests_seconds_count[30d]))
) * 100

# Remaining error budget (if SLO is 99.9%)
(
  (
    sum(rate(http_server_requests_seconds_count{status!~"