stream-aggregation-helper

What this skill does

# Stream Aggregation Helper

Design, deploy, and verify [stream aggregation](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/) rules
for VictoriaMetrics. Stream aggregation reduces cardinality, sample frequency, or query cost by aggregating samples
**by time and by labels** at `vmagent` (or single-node vmsingle) **before** they reach storage.

The skill walks through five phases:

1. **Gate** — is stream aggregation the right tool for this problem?
2. **Intake** — current cardinality, collection topology, what consumers need.
3. **Decide** — where in the pipeline, interval, output function, `by`/`without`, scaling.
4. **Config** — produce a ready-to-apply YAML plus a phased rollout plan.
5. **Verify** — queries and `vm_streamaggr_*` metrics that confirm the rule is doing what was intended.

## Environment

Uses the same env vars as the `victoriametrics-query` skill. The skill works **without** these (interview-only
mode) but is much more accurate when it can query the user's VictoriaMetrics directly.

```bash
# $VM_METRICS_URL - base URL (optional)
#   cluster: export VM_METRICS_URL="https://vmselect.example.com/select/0/prometheus"
#   single:  export VM_METRICS_URL="http://localhost:8428"
# $VM_AUTH_HEADER - full HTTP header line (empty if no auth)
#   Prod:  export VM_AUTH_HEADER="Authorization: Bearer <token>"
#   Local: export VM_AUTH_HEADER=""
```

---

## Phase 1: Is stream aggregation the right tool?

Stream aggregation is the right answer when **all** of these hold:

- For every output series that the rule will produce, **all** input samples that contribute to it
  land on a single aggregator process. This holds in three common topologies:
  1. A single `vmagent` (or vmsingle) sees every sample for the matched series.
  2. Multiple vmagents shard traffic by some label (e.g., `instance`, `kubernetes_namespace`,
     `__name__`) and the aggregation **preserves that shard key** — see the shard-key invariant in
     Phase 3a (case F). Tiered topologies like *tier-1 vmagents shard by `instance` → tier-2
     vmagents aggregate within `instance`* are explicitly supported.
  3. HA replicas all see every sample **and** the aggregation output is dedup-safe (gauge-style:
     `min`, `max`, `avg`, `rate_*`, `increase*`, `last`, `quantiles`, etc.). Counter-style outputs
     (`total`, `total_prometheus`, `histogram_bucket`) cannot be HA-aggregated because each replica
     maintains independent monotonic state. See the High Availability section.
- The user is willing to lose per-source resolution (per-pod, per-instance) for the matched metric,
  OR only wants to drop a high-cardinality label they don't actually consume.
- Consumers (dashboards, alerts, recording rules) either don't need the dropped dimensions, or can
  be migrated to the aggregated metric name.
- State loss on `vmagent` restart is acceptable. Aggregation state is in-memory; on restart the
  first one or two intervals are discarded by default.

**Shard-key invariant**: an output series is correct only if every input sample for it goes to the
same aggregator. Translated to rules: the labels used to shard the stream must appear in `by:` (or
must not appear in `without:`) of every aggregation rule downstream of the shard. Drop them and you
get partial aggregates per shard — silently wrong.

If any of these fail, redirect the user:

| Situation | Better tool |
|-----------|-------------|
| Need to drop entire unused metrics | `metric_relabel_configs` with `action: drop` — see `victoriametrics-unused-metrics-analysis` |
| Need to drop labels without time-bucketing | `metric_relabel_configs` with `action: labeldrop`, or `-streamAggr.dropInputLabels` |
| Need historical downsampling on existing data | vmstorage downsampling (`-downsampling.period`) |
| Want a derived metric for queries, not ingestion | Recording rules in vmalert |
| Single-node setup but stream-aggregating before storage | `-streamAggr.config` on vmsingle is fine — same engine; just say so |
| Behind a load balancer that splits the stream (round-robin) | Shard traffic deterministically by labels (`-remoteWrite.shardByURL.labels`), then aggregate while preserving the shard key. Round-robin LBs split a single output series across replicas → incomplete results. See [Common Mistakes › Behind load balancer](https://docs.victoriametrics.com/victoriametrics/stream-aggregation/#put-aggregator-behind-load-balancer) |
| Tiered vmagents (tier-1 shards, tier-2 aggregates) | Supported and common at scale. Aggregation rules on tier-2 must **keep the shard key** in `by:` (or omit it from `without:`). See case F below. |

If the gate passes, continue. If not, recommend the alternative and stop.

---

## Phase 2: Intake

Collect three groups of facts. Run the live queries in parallel **only if** `VM_METRICS_URL` is set; otherwise
ask the user the same questions directly.

### 2a. Cardinality picture

What metric/set is the problem? How many series, driven by which labels?

```bash
# Top series by metric name (find the offenders)
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=20" | \
  jq '.data | {totalSeries, seriesCountByMetricName, seriesCountByLabelName}'

# For each suspect metric, see unique values per label
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=20&match[]=METRIC_NAME" | \
  jq '.data.labelValueCountByLabelName'
```

**Measure post-aggregation cardinality directly.** Once you have a candidate `by:` or `without:`,
run the equivalent grouping against current data instead of multiplying per-label unique-value
counts. The product-of-labels formula assumes every Cartesian combination of label values exists,
which is rarely true — and worse, can be **over-optimistic** when there are correlated labels or
labels you forgot to enumerate. The measured count is authoritative:

```bash
# Post-aggregation series count, using the labels you plan to KEEP.
# Replace <metric> and <kept_labels>. For counters use rate(...[5m]); for gauges use last_over_time.
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  --data-urlencode 'query=count(sum(rate(<metric>[5m])) by (<kept_labels>))' \
  "$VM_METRICS_URL/api/v1/query" | jq '.data.result'

# Equivalent form using the labels you plan to DROP:
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  --data-urlencode 'query=count(sum(rate(<metric>[5m])) without (<dropped_labels>))' \
  "$VM_METRICS_URL/api/v1/query" | jq '.data.result'
```

This gives the actual number of output series the rule will produce *on the current data*, which is
the right basis for the savings estimate in Phase 4d. Run it for each candidate rule before
committing to a config.

Interview questions if live queries are unavailable:

- Which metric or metric pattern is the problem?
- Approximate total series count for it?
- Which labels exist on it, and roughly how many unique values for each?
- Counter, gauge, or histogram?
- Scrape interval / push frequency?

When live queries aren't possible, treat the product-of-labels formula as an **upper bound** only,
and note explicitly to the user that the real post-aggregation count must be verified once data is
reachable.

### 2b. Collection topology

How do samples flow today? Sketch one of these:

```
A: app  -> vmagent -> remote write -> TimeSeries DB         (vmagent already in path)
B: app  -> remote write -> TimeSeries DB                    (no vmagent — must insert one)
C: app  -> [Prometheus | OTel | StatsD-exporter] -> remote write -> TimeSeries DB
D: app  -> load balancer -> N vmagents -> TimeSeries DB     (LB splits stream — dangerous for SA)
E: app  -> vmagent (HA pair / N replicas) -> TimeSeries DB  (multi-vmagent — dedup needed)
F: app  -> vmagent cluster (sharded) -> TimeSeries DB       (sharded by `-remoteWrite.shardByURL`)
```

The destination is shown as a generic "TimeSeries DB" — stre