grafana-panel-engineer

What this skill does

# Grafana Panel Engineer

Design Grafana dashboards that load fast, answer the right question in under 30 seconds, and survive being looked at by a sleepy on-call engineer on a phone. Acts as a senior SRE who has built and pruned thousands of dashboards across Prometheus, Loki, Tempo, Pyroscope, Mimir, InfluxDB, and CloudWatch — and knows which dashboard layouts the rotation actually opens during incidents (RED for service health, USE for capacity, customer journey for product-tier impact).

This skill builds dashboards. It does not write your detection logic, define your SLOs, or replace your APM. It assumes data is already flowing into a Grafana datasource; the job is panel selection, layout, variables, drill-downs, exemplar linking, and query optimization. Output is dashboard JSON (importable, version-controlled, ideally as Grafana provisioning YAML or Terraform `grafana_dashboard`), plus a panel-design playbook for new services.

## Usage

Invoke when:

- A new service is shipping and needs its dashboard pack from day one
- An existing dashboard takes 60 seconds to load
- Engineers say "I never know which dashboard to open during an incident"
- A panel uses a query with cardinality of 50,000 series and the renderer hangs
- Exemplars from Prometheus aren't wired to Tempo traces
- Variable dropdowns produce 10,000 entries; selection is unusable
- Dashboards are duplicated across teams with subtle drift
- A dashboard alert triggers but the panel doesn't show what tripped it
- Loki + Prometheus + Tempo are all installed but never queried in the same panel
- Grafana usage analytics show the dashboard count is 1,200 but only 80 are opened in a month

**Basic invocations:**
> Build a dashboard pack for our new T0 checkout service
> Audit our 1,200 dashboards and prune the unused ones
> Wire Prometheus exemplars to Tempo traces in our HTTP latency panel
> Convert our service's dashboard from USE to RED + Four Golden Signals
> Optimize this dashboard — it takes 90s to load and hangs the browser

## Inputs Required

- Grafana URL + API key (or service-account token, ideally read-write)
- Datasource list: Prometheus/Mimir/Thanos, Loki, Tempo, Pyroscope, others
- Service catalog: name, tier (T0-T3), team owner, runbook URL
- Existing dashboard inventory (export via API: `GET /api/search?type=dash-db`)
- Metric naming conventions (Prometheus: `service_name_thing_total`, etc.)
- Trace sampling: % sampled, exemplar storage backend, trace ID propagation
- SLOs and the metrics behind them (for SLO dashboards)
- Constraints: max series per panel, query timeout, dashboard refresh rate

## Workflow

1. **Inventory existing dashboards.** API: `GET /api/search`, `GET /api/dashboards/uid/{uid}`. Tag each with: last viewed (Grafana usage analytics), data sources used, panel count, owner. Anything not viewed in 90 days and not the only copy of its data is a deletion candidate.

2. **Classify dashboards by purpose.** Service health (RED), capacity (USE), customer journey (funnel/conversion), SLO (error budget), debug (drill-down for incident response), executive (summary). Each purpose has a different layout pattern.

3. **Pick a layout pattern per dashboard.** USE (Utilization/Saturation/Errors) for resource panels — CPU, memory, disk, network. RED (Rate/Errors/Duration) for request-driven services. Four Golden Signals for full service health. Customer journey for product flows. See Dashboard Layouts.

4. **Pick the right panel type per metric.** Time series for trends, stat for current value with trend sparkline, gauge for "% of capacity" only, heatmap for distributions and percentile streaks, table for top-N, state-timeline for service state, bar gauge for ranked comparison. See Panel Type Decision Tree.

5. **Design variables (templating).** Variables drive reusability. Cascade environment → cluster → namespace → service → instance. Use `Multi-value` and `Include All` carefully — they explode query cardinality. See Variable Templating Patterns.

6. **Build the queries.** PromQL with `rate()` for counters, `histogram_quantile()` for histograms, `irate()` only for short-window debug. Set query interval to match Prometheus scrape (`$__rate_interval`), not the dashboard refresh.

7. **Add drill-down links.** Each panel that shows a metric for a service should link to the deeper service dashboard. Each error-rate panel should link to logs (Loki) and traces (Tempo) filtered to the same time range and labels.

8. **Wire exemplars.** Prometheus histograms with exemplar support emit trace IDs alongside latency buckets. Configure the panel's "Exemplars" toggle and link to Tempo with `${__value.raw}` as the trace ID.

9. **Optimize for performance.** Limit cardinality (`topk`, `bottomk`), set max points (`Max data points` panel option), use recording rules for expensive queries, push aggregation to Prometheus where possible. See Performance Optimization.

10. **Set dashboard refresh + time range defaults.** Refresh: 30s for live ops dashboards, 5m for everything else, off for debug dashboards. Default time range: last 1h for ops, last 24h for trend dashboards.

11. **Add panel descriptions.** Every panel has a description: what the metric means, what's normal, what's bad, link to runbook. Right-click info icon shows it.

12. **Generate alerts from panels.** Grafana unified alerting: from a panel, build an alert rule with the same query. Tie alert annotations to dashboard panel link so on-call gets a deep-link to the panel.

13. **Provision via code.** Dashboard JSON in git, deployed via Grafana provisioning YAML or Terraform. UI edits drift; provisioning prevents drift.

14. **Schedule audits.** Monthly: stale dashboard pruning (Grafana usage analytics), variable cardinality audit, query cost review. Quarterly: layout refresh, datasource audit.

## Dashboard Layouts

### USE Method (Brendan Gregg)

For *resources*: CPU, memory, disk, network, file handles. Three panels per resource.

```
[ Utilization % ]   [ Saturation (queue depth, runqueue) ]   [ Errors / sec ]
```

Layout: one row per resource (CPU row, Memory row, Disk row, Network row). Each row has 3 panels in U/S/E order. Time series, stacked by host or pod, top-10 by `topk(10, ...)`.

**When to use:** capacity / fleet health dashboards. Not for service health (use RED).

### RED Method (Tom Wilkie)

For *services*: Rate (req/s), Errors (errors/s or %), Duration (p50/p95/p99).

```
Row 1 — Top-line:
  [ Rate (req/s) ]   [ Error rate (%) ]   [ p99 Duration (ms) ]
Row 2 — Per-endpoint:
  [ Rate by endpoint (stacked) ]   [ Error rate by endpoint (table top-10) ]   [ Duration heatmap ]
Row 3 — Per-status:
  [ Status code breakdown (stacked area) ]   [ 5xx by endpoint ]   [ Slow endpoints (table) ]
```

**When to use:** any request-driven service (HTTP, gRPC, message queue consumer). Default for T0/T1 services.

### Four Golden Signals (Google SRE Book)

Latency, Traffic, Errors, Saturation. RED + saturation.

```
Row 1: [ Traffic (req/s) ] [ Errors (%) ] [ Latency p99 ] [ Saturation (CPU/mem) ]
Row 2: [ Traffic by version (deploy overlay) ] [ Errors by error class ] [ Latency heatmap with exemplars ] [ Saturation forecast ]
Row 3: [ Throughput vs latency scatter ] [ Top errors (table) ] [ Slow queries (table from logs) ] [ Capacity headroom % ]
```

**When to use:** full service health for T0 services. Most flexible single dashboard.

### Customer Journey Layout

For end-to-end product flows (signup, checkout, search-to-purchase). Each panel = one step in the funnel.

```
Row 1: [ Step 1 success% ] [ Step 2 success% ] [ Step 3 success% ] [ Step N success% ]
Row 2: [ Step 1 latency p99 ] [ Step 2 latency p99 ] ...
Row 3: [ Drop-off Sankey or stacked bar ]   [ Bottleneck step over time ]
Row 4: [ Per-segment funnel (enterprise vs free) ]   [ Errors at top drop-off step ]
```

**When to use:** product-tier dashboards, exec dashboards, SRE dashboards for revenue-path services.

### SLO / Error Budget Layout

```
Row 1 (top-li