promql

What this skill does

# PromQL

PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an
expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant
vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them
fast.

Authoritative reference for **writing PromQL queries** — dashboards, alerts, recording rules, exploration. The companion
`backend/prometheus` skill covers **instrumentation** (metric types, naming, labels, exporters). When both apply, use
`backend/prometheus` for emitting metrics and this skill for querying them.

## References

- **Selectors and data types** — [`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`] Expression types, instant and
  range vector selectors, label matchers (`=`, `!=`, `=~`, `!~`), time durations, offset and `@` modifiers, staleness,
  instant vs range queries, string and float literals
- **Operators** — [`${CLAUDE_SKILL_DIR}/references/operators.md`] Arithmetic, comparison, logical/set, vector matching
  (`on`, `ignoring`, `group_left`, `group_right`), aggregation operators, `by`/`without`, fill modifiers, operator
  precedence
- **Functions** — [`${CLAUDE_SKILL_DIR}/references/functions.md`] Full function catalog: counter family, gauge
  functions, classic and native histograms, `*_over_time`, existence checks, label manipulation, math, time, type
  conversion, sorting, trigonometric
- **Native histograms** — [`${CLAUDE_SKILL_DIR}/references/native-histograms.md`] Native vs classic detection,
  `histogram_quantile` across both forms, native-only functions (`histogram_avg`, `histogram_count`, `histogram_sum`,
  `histogram_fraction`, `histogram_stddev`, `histogram_stdvar`), aggregation without `le`, NHCB semantics, trim
  operators, migration patterns, function compatibility matrix, gotchas, annotations
- **Subqueries** — [`${CLAUDE_SKILL_DIR}/references/subqueries.md`] Syntax, resolution and alignment, nested subqueries,
  when to use subqueries vs recording rules, pitfalls
- **Optimization and pitfalls** — [`${CLAUDE_SKILL_DIR}/references/optimization.md`] Cardinality awareness, common
  pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets,
  staleness, rate window sizing, native histograms, diagnostics
- **Recording and alerting rules** — [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`] Rule file structure,
  `level:metric:operations` naming, ratio aggregation patterns, alert syntax, `for`/`keep_firing_for`, templating,
  alerting best practices, anti-patterns

Read the relevant reference before proceeding.

---

## Data Types

Every expression has one type. Functions and operators require specific input types — type mismatch is the most common
cause of "query parses but returns nothing".

- **instant vector** — set of series, one sample per series at one timestamp. Default result of metric selectors,
  aggregations, most functions.
- **range vector** — set of series, multiple samples per series across a time window. Produced **only** by range vector
  selectors (`metric[5m]`) and subqueries. Cannot be graphed directly — must pass through a function returning an
  instant vector.
- **scalar** — single numeric value, no labels. Used as numeric arguments and in arithmetic.
- **string** — function arguments only; cannot be a query result.

**Function type rules:**

- `rate`, `irate`, `increase`, `delta`, `idelta`, `deriv`, `predict_linear`, `*_over_time` — range vector → instant
  vector
- `histogram_quantile`, `histogram_fraction`, `histogram_*` — instant vector (histogram or bucket series) → instant
  vector
- Aggregation operators (`sum`, `avg`, `max`, `topk`, ...) — instant vector → instant vector
- `scalar()` — instant vector → scalar; `vector()` — scalar → instant vector

## Selectors

### Instant Vector Selectors

```promql
http_requests_total                                # all series with this metric name
http_requests_total{job="api"}                     # filtered by label
http_requests_total{job="api", method="GET"}       # multiple matchers — AND
http_requests_total{status=~"5.."}                 # regex match (RE2, fully anchored)
http_requests_total{method!="OPTIONS"}             # negative match
{__name__=~"http_.*"}                              # regex on metric name via __name__
```

**Label matchers:** `=` exact, `!=` not equal, `=~` regex match, `!~` negative regex.

**Regex is fully anchored** — `=~"foo"` matches `^foo$`, not arbitrary substrings. Use `=~".*foo.*"` for substrings.

**Required selectors:** at least one matcher must not match the empty string. `{job=~".*"}` is illegal; `{job=~".+"}` is
valid.

### Range Vector Selectors

Append `[duration]` to a selector:

```promql
http_requests_total[5m]                            # last 5 minutes of samples
http_requests_total{job="api"}[1h]
```

Duration units: `ms`, `s`, `m`, `h`, `d`, `w`, `y`. Combine longest-first: `1h30m`, `12h34m56s`. Floats with units are
not allowed (`1.5h` is invalid — use `1h30m`).

### Modifiers

- **`offset <duration>`** — time-shift into the past (or future with negative). Must immediately follow the selector.

  ```promql
  http_requests_total offset 5m
  rate(http_requests_total[5m] offset 1w)            # rate one week ago
  ```

- **`@ <unix-timestamp>`** — pin evaluation to an absolute time. `start()` and `end()` resolve to the range query's
  bounds.

  ```promql
  http_requests_total @ 1609746000
  rate(http_requests_total[5m] @ end())
  ```

**Staleness:** instant vector selectors return the most recent sample within the lookback window (default 5 minutes).
Series with no sample in the window disappear from the result.

Full selector reference, lookback delta tuning, instant-vs-range-query semantics:
[`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`].

## Operators

### Arithmetic and Comparison

`+`, `-`, `*`, `/`, `%`, `^` — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.

`==`, `!=`, `>`, `<`, `>=`, `<=` — by default **filter**: drop series where the comparison is false. Add `bool` after
the operator to return `0` or `1` instead.

```promql
http_requests_total > 100                          # only series whose latest value exceeds 100
http_requests_total > bool 100                     # all series, value is 1 if > 100 else 0
```

### Logical/Set Operators

Operate on label-set membership, ignoring sample values:

- `vector1 and vector2` — intersection
- `vector1 or vector2` — union
- `vector1 unless vector2` — complement

```promql
up{job="prom"} or up{job="node"}                       # union
metric and on(device) (other_metric == 0)              # constrained intersection
```

### Vector Matching

The biggest source of "no results returned". Default rule: **two elements match if they have exactly the same label
set** (after the metric name drops). When operands have different label sets, match keywords are required:

- **`on(labels)`** — match only on these labels; ignore all others
- **`ignoring(labels)`** — match on all labels except these

**Group modifiers for many-to-one** (arithmetic, comparison, trigonometric only — not set operators):

- **`group_left(extra_labels)`** — many on the left, one on the right; copy extra_labels from right to result
- **`group_right(extra_labels)`** — symmetric

**Canonical patterns:**

```promql
# Error ratio: errors carry an extra `code` label; ignore it to match
errors:rate5m{code="500"} / ignoring(code) requests:rate5m

# Many-to-one: same denominator for every code
errors:rate5m / ignoring(code) group_left requests:rate5m

# Join info-style metric: bring `version` label into result
node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info
```

### Aggregation

Instant vector → instant vector with fewer elements.

- **`sum`**, **`avg`**, **`min`**, **`max`** — re