loki-label-analyzer

What this skill does

# Loki Label Strategy Evaluator

You are an expert in Grafana Loki label strategy. When asked to evaluate, audit, design, or improve a Loki label strategy — or when a user asks why their Loki queries are slow — use this guide to provide structured, actionable advice.

---

## Core Concepts

**Streams** are the fundamental unit in Loki. Each unique combination of label key-value pairs creates a new stream. Too many streams = performance problems. Too few = broad, slow queries.

**Cardinality** = the number of unique values a label can have. High-cardinality labels (like `pod`, `user_id`, `request_id`) dramatically increase stream count and hurt performance — *especially* when those labels are not specified in every query.

**The dual impact rule**: High-cardinality labels hurt on both paths:
- **Ingestion path**: More streams → larger index, higher storage costs
- **Query path**: If a high-cardinality label exists but isn't in the query selector, Loki must scan ALL streams matching the other selectors — catastrophic for performance

**The key question for any dynamic label**: "Will this label be used in 9 out of 10 queries?" If no → it should NOT be a label.

---

## Label Evaluation Framework

When auditing a label strategy, assess each label against these criteria.

### Cardinality Scoring

| Label Example | Cardinality | Verdict |
|---|---|---|
| `env` (prod/staging/dev) | 2–5 values | ✅ Good |
| `level` (info/warn/error) | 3–6 values | ✅ Good |
| `namespace` (K8s) | Tens | ✅ Acceptable |
| `instance` / `hostname` | Hundreds–thousands | ⚠️ Evaluate access patterns |
| `pod` | Thousands + transient | ❌ Avoid as label |
| `user_id`, `request_id` | Unbounded | ❌ Never use as label |

### Access Pattern Alignment
For each label, ask:
- Is this label used as a selector in most queries targeting these logs?
- Does this label logically segment data in the way users think about it?
- Would removing this label force users to scan dramatically more data?

### Static vs. Dynamic Label Values
- **Static labels** (values don't change per log line, e.g., `platform=linux`, `job=agent`) add no cardinality cost relative to the query scope. Use freely for LBAC, exploration, and alert routing.
- **Dynamic labels** (values change per log line) must be bounded. Keep possible values in the single digits or low tens.

### Consistency Check
- Are label names consistent across services? (case-sensitive — `Level` ≠ `level`)
- Are label values normalized? (`INFO`, `info`, `Info` should all become `info`)
- Is there a naming convention? (pick one: `snake_case` or `camelCase` — be consistent)

---

## Evaluation Output Format

When auditing a label set, produce a report in this structure:

```
## Loki Label Strategy Audit

### Summary
[1-2 sentence overall assessment]

### Label Analysis
| Label | Cardinality | Used in Queries? | Verdict | Action |
|---|---|---|---|---|
| app | Low (tens) | Always | ✅ Keep | — |
| pod | Very High (transient)| Rarely | ❌ Remove | Move to structured metadata or embed in log line |

### Estimated Impact
- Stream count reduction: [X streams → Y streams]
- Query performance: [describe improvement]
- Storage impact: [if log line changes are involved]

### Recommended Label Set
[Final recommended labels]

### Migration Notes
[How to implement changes via Alloy/Agent pipeline stages]
```

---

## Recommended Common Labels

Every log source should consider these base labels — all low cardinality, high query value:

| Label | Purpose |
|---|---|
| `app` / `service` | Identifying the generating application |
| `env` | Environment (prod, staging, dev) |
| `cluster` | Multi-cluster differentiation |
| `region` | Geographic region |
| `level` | Log severity — normalize to: `info`, `warn`, `error`, `debug` |
| `job` | Collector job name |
| `team` / `squad` | Ownership (also useful for LBAC) |
| `source` | Log origin type (`file`, `k8s-events`, `journal`, `syslog`, etc.) |
| `classification` | Data sensitivity level — for LBAC policies |

---

## Kubernetes Pod Logs

### Recommended Labels

| Label | Description |
|---|---|
| `namespace` | K8s namespace — delineates isolation boundaries |
| `container` | Container name — low cardinality, differentiates log formats |
| `service` | K8s service generating logs |
| `workload` | `{controller_kind}/{controller_name}` e.g. `ReplicaSet/payment-api` — **strongly recommended** |

**Why `workload` beats `app` for K8s**: Derived from `{{controller_kind}}/{{controller_name}}` — static values that never change like pod names do. Unlike `app` (which may aggregate multiple workload types), `workload` is precise and predictable. Users always know exactly what value to query.

### Labels to AVOID in Kubernetes

**`pod` label** ❌
- Highly transient: pod names change on every restart/rollout
- Very high cardinality: 5 pods × 2 containers = 10 streams; add `pod` → 10 × N streams
- Users almost never query for a specific pod; they query for the *workload*
- **Solution**: Use `workload` as the label; store `pod` in structured metadata or embed in the log line

**`filename` label (raw K8s path)** ❌
- K8s log paths contain pod UID: `/var/log/pods/{namespace}_{pod}_{pod_id}/{container}/{rotation}.log`
- The `pod_id` component makes this unbounded
- **Solution**: Normalize to `/var/log/pods/{namespace}/{controller_name}/{container}.log` or drop entirely

```alloy
// Normalize K8s filename to remove pod UID
stage.replace {
 source = "filename"
 expression = "/var/log/pods/([^/]+)_[^_]+_[^/]+/([^/]+)/\\d+\\.log"
 replace = "/var/log/pods/$1/$2/current.log"
}
```

---

## Host / VM / Bare Metal Labels

In addition to common labels, add:

| Label | Description | Notes |
|---|---|---|
| `instance` | Hostname of the machine | Cardinality = number of machines; acceptable for fixed infrastructure |
| `filename` | Full path to the file being tailed | Normalize rotating filenames — strip date suffixes |

```alloy
// Remove date suffixes from rotating log file names
// /var/log/myapp/logfile-20230927.txt → /var/log/myapp/logfile.txt
stage.replace {
 source = "filename"
 expression = "-\\d{8}(\\.log|\\.txt)$"
 replace = "$1"
}
```

---

## Journal Logs

When collecting via `loki.source.journal`, many labels are auto-discovered under `__journal__*`:
`boot_id`, `cap_effective`, `cmdline`, `comm`, `exe`, `gid`, `hostname`, `machine_id`, `pid`, `stream_id`, `systemd_cgroup`, `systemd_invocation_id`, `systemd_slice`, `systemd_unit`, `transport`, `uid`

Almost all are high-cardinality. **Keep only**:
- `instance` — hostname where journal logs were collected
- `unit` — the `systemd_unit` name (e.g., `nginx.service`)

Drop everything else:
```alloy
loki.process "journal_labels" {
 forward_to = [...]
 stage.label_keep {
 values = ["instance", "unit", "env", "cluster"]
 }
}
```

---

## Structured Metadata

Structured metadata attaches key-value pairs to log entries *without* making them index labels. The ideal home for high-cardinality values users occasionally need.

**Requires**: Loki 2.9+, Grafana Agent/Alloy. Enable via `limits_config`:
```yaml
limits_config:
 allow_structured_metadata: true
```

**Good candidates for structured metadata** (not labels):
- `pod` — K8s pod name
- `node` — K8s worker node
- `version` / `image` / `tag`
- `trace_id` / `user_id`
- `process_id`
- `restarted` — pod restart timestamp

Query structured metadata at query time without a parser:
```logql
{app="payment-api"} | pod="payment-api-7f9d4b-xk2r9"
```

---

## Embedding Metadata in Log Lines

When structured metadata isn't available, embed high-cardinality values into the log line rather than using them as labels.

### Method 1: stage.template (append to log line)

```alloy
loki.process "embed_pod" {
 forward_to = [...]

 // For JSON logs
 stage.match {
 selector = "{} |~ \"^\\s*\\{\""
 stage.replace {
 expression = "\\}$"
 replace = ""
 }
 stage.template {
 source = "log_line"
 template = "{{ .Entry }},\"_pod\":\"{{ .pod }}\"}"
 }
 }

 // For text logs
 stage.match {