prometheus-cardinality-troubleshooter

What this skill does

# Prometheus Cardinality Troubleshooter

You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.

This skill is **diagnostic and operational**. For schema design and prevention, route to `prometheus-label-strategy`.

---

## Before You Remediate: The One Rule

Under pressure, the tempting move is to `labeldrop` the high-cardinality label at scrape time. **Do not.** You cannot remove, at scrape time, any label that makes a series unique — not `pod`, not `instance`, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually **breaks the data**:

- Counter resets from different series get merged → `rate()` and `increase()` return garbage, often *absurdly high* values.
- Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and **inflated** DPM, not reduced.
- The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks "why is my DPM so high / why is `rate()` absurd?" and there's nothing to point to.

The only safe remediations are:

1. **Drop an *entire* unwanted metric** (`action: drop` on `__name__`) — you're discarding the whole metric, not merging distinct series.
2. **Fix the source** — stop the application emitting the bad label (the real fix for unbounded `path`, `user_id`, etc.).
3. **Adaptive Metrics** — for structural cardinality on series you can't fix at the source. It aggregates *correctly* (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like `pod`. Route to `adaptive-metrics`.

Everywhere below that says "drop a label," read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never `labeldrop` a distinguishing label.

---

## Symptom → Likely Cause

| Symptom | Likely Cause | First Action |
|---|---|---|
| Prometheus OOMKilled or memory growing linearly | Active series growth (often from a new bad metric or label) | [Active Series triage](#step-1-active-series-triage) |
| Single PromQL query slow or OOMs the querier | One or more metrics in the query have high cardinality | [Per-query drill-down](#step-3-per-metric-drill-down) |
| Remote write lagging, WAL growing | Sample throughput spike — series count OR scrape interval changed | [Active Series triage](#step-1-active-series-triage) + check scrape intervals |
| `429 Too Many Samples` / `out of bounds` errors | Hitting Mimir/Cortex ingester per-tenant series limit | [Per-metric drill-down](#step-3-per-metric-drill-down), find the new offender |
| Grafana Cloud Active Series bill spiked | New metric, new label, or rollout creating churn | [Per-metric drill-down](#step-3-per-metric-drill-down) + churn check |
| Grafana Cloud DPM bill spiked but Active Series flat | Scrape interval shortened, OR remote_write sending duplicates | DPM-side issue — route to `dpm-finder` |
| `series_limit_per_user` errors after a deploy | Application change introduced a new bad label | [Recent change diff](#step-4-recent-change-diff) |
| Series count grows then resets every restart | Series churn from ephemeral label values | [Churn diagnosis](#step-5-churn-diagnosis) |

---

## Step 1: Active Series Triage

### Get the headline number

```promql
# Total active series in the local Prometheus
prometheus_tsdb_head_series

# Or for Mimir / Grafana Cloud Metrics (per tenant)
cortex_ingester_memory_series{user="<tenant>"}
```

Compare to recent history:
```promql
# Growth over the last 7 days
deriv(prometheus_tsdb_head_series[7d]) * 86400
```

A growth rate > a few % per day on a stable application set is a red flag.

### Use the TSDB status endpoint

Prometheus exposes a built-in cardinality breakdown:

```bash
curl -s http://prometheus:9090/api/v1/status/tsdb | jq
```

Returns:
- `seriesCountByMetricName` — top metrics by series count
- `labelValueCountByLabelName` — top labels by unique value count
- `memoryInBytesByLabelName` — top labels by memory footprint
- `seriesCountByLabelValuePair` — top label-value pairs by series count

This is usually the fastest path to "which metric / which label is the problem."

For Grafana Cloud:
```bash
# Same endpoint, authenticated against the per-tenant Mimir
curl -s -u "<user>:<token>" \
  "https://prometheus-prod-XX.grafana.net/api/prom/api/v1/status/tsdb" | jq
```

---

## Step 2: Read the Output

### Top metrics by series count

```json
"seriesCountByMetricName": [
  { "name": "http_request_duration_seconds_bucket", "value": 184320 },
  { "name": "go_gc_duration_seconds",               "value": 80 },
  ...
]
```

**Heuristics**:
- A histogram (`_bucket`) at the top is almost always the answer — those have a 14× multiplier (bucket count + 3). The fix is usually **reducing the labels on the underlying histogram at the source** (in instrumentation code), not stripping them at scrape and not touching the buckets themselves.
- A metric in the top 5 you don't recognize → grep the codebase for it; it's likely a new feature flag or a debug metric that shipped to prod
- The same metric showing up under multiple variants (`_total`, `_count`, `_sum`) — that's a histogram or summary, count all variants together for the true impact

### Top labels by unique value count

```json
"labelValueCountByLabelName": [
  { "name": "url",       "value": 84210 },
  { "name": "trace_id",  "value": 41000 },
  { "name": "pod",       "value": 1820 }
]
```

**Red flags**:
- Any label with >10K unique values is almost certainly a bug. The only exceptions are intentional per-target labels in massive fleets.
- `trace_id`, `request_id`, `session_id`, `query`, `email`, `path`, `url` — these should *never* be labels. They belong in exemplars, logs, or traces.
- `pod` with thousands of values — see [Churn diagnosis](#step-5-churn-diagnosis); recent churn often inflates this number

---

## Step 3: Per-Metric Drill-Down

Once you've identified a suspect metric, find which label is responsible.

### Count distinct label values per label, for one metric

```promql
# How many unique values does each label have on this metric?
count by (__name__) (
  count by (__name__, label_name_here) (
    http_request_duration_seconds_bucket
  )
)
```

Repeat per label, or use the helper:

```bash
# Via the Prometheus HTTP API
curl -s "http://prometheus:9090/api/v1/labels?match[]=http_request_duration_seconds_bucket" | jq -r '.data[]' | \
  while read label; do
    count=$(curl -s "http://prometheus:9090/api/v1/label/${label}/values?match[]=http_request_duration_seconds_bucket" | jq '.data | length')
    echo "${count}  ${label}"
  done | sort -rn | head -20
```

### Find the top label values for one label

```promql
# Top 20 path values for http_requests_total
topk(20,
  count by (path) (http_requests_total)
)
```

If you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.

### Per-metric series count, grouped

```promql
# Series-per-instance breakdown — if uneven, one instance is misbehaving
sum by (job, instance) ({__name__=~"my_metric.*"})
```

---

## Step 4: Recent Change Diff

If the cardinality fire started recently, the cause is almost always a recent change. Diff what's there now against what was there before.

### List of metrics, current vs. yesterday

Via Grafana Cloud cardinality dashboard, or:
```promql
# Current metrics
group by (__name__) ({__name__!=""})

# Compare to last week (offset)
group by (__name__) ({__name__!=""} offset 7d)
```

Diff externally. A new metric near the top of `seriesCountByMetricName` that wasn't there a week ago → that's your offender.

### Correlate with deploys

```promql
# Active series correlated with build_info
prometheus_tsdb_head_series
# Overlay with:
changes(app_build_info[1d])
```

A vertical step in series count aligned with a