signoz-setting-up-observability

What this skill does

# Setting Up Observability After Ingestion

Take a service from "telemetry is flowing into SigNoz" to "I have a
dashboard, alerts, saved views, and a tuning loop." This skill is an
**orchestration layer, not a standalone reference** — the mechanical
work lives in the sibling SigNoz skills and the MCP tools/resources hold
the current payload schemas. This skill sequences them and supplies the
judgment calls (scope, SLOs, thresholds, what to skip) that no single
skill owns; it deliberately does not restate payload schemas or field
rules, so read the relevant `signoz://…` resource before composing any
payload.

Use this when traces, logs, or metrics are **already landing** in
SigNoz and you want an opinionated, SLO-aware setup — not when you're
still wiring up the SDK. Audience: two consumers — an autonomous AI SRE
agent that runs without a human in the loop (e.g. an in-product SigNoz
assistant), and an engineer at a Claude Code / Codex / Cursor prompt.
Both follow the same flow; where a step needs a human decision (Phase 1
triage, Phase 5 sign-off), the host decides how to surface it, and an
autonomous host fills the gap from upstream context instead of blocking.

## Prerequisites

This skill calls SigNoz MCP server tools (`signoz:signoz_list_services`,
`signoz:signoz_list_dashboards`, `signoz:signoz_get_field_keys`,
`signoz:signoz_execute_builder_query`, `signoz:signoz_create_dashboard`,
`signoz:signoz_create_alert`, `signoz:signoz_create_view`, etc.) and
delegates to sibling skills. Before running the workflow, confirm the
`signoz:signoz_*` tools are available. If they are not, the SigNoz MCP
server is not installed or configured — run `signoz-mcp-setup` first. Do
not fall back to raw HTTP calls or fabricate payloads.

## When to use

Use this skill when the user wants the **whole post-ingestion setup**:
SLI/SLO → exploration → dashboard → views → alerts → tuning, in any
combination of those deliverables, sequenced as one workflow.

Do NOT use this skill when the user wants a single artifact — hand off
directly:
- Just a dashboard → `signoz-creating-dashboards`.
- Just one static / threshold alert or notification rule →
  `signoz-creating-alerts`. (But SLO / burn-rate / error-budget
  alerting — even with no dashboard — stays *here*: it needs the SLI/SLO
  capture and burn-rate judgment in the phases below.)
- Just a saved Explorer view → `signoz-managing-views`.
- A one-off exploratory query → `signoz-generating-queries`.
- A concept/doc lookup (SRE methods, OTel conventions, burn-rate math)
  → `signoz-searching-docs` / `signoz:signoz_search_docs`.

Each phase below points to the skill that does the mechanical work; this
skill owns the order and the decisions.

## Phase 1 — Triage scope (don't skip)

Before touching any tools, get these answers. Each unblocks a downstream
phase.

1. **What does this service do, and does it call external services?**
   One or two sentences. This shapes the SLI (what "working" means) and
   flags downstream dependencies worth monitoring — if it calls a DB,
   cache, queue, or third-party API, plan to watch the client-span
   latency and error rate to each dependency, since a healthy service
   fronting a sick dependency still fails users.
2. **What deliverables?** dashboard / alerts / saved views — any
   combination.
3. **Infra monitoring too? (optional — ask)** If the service runs on
   infra you own (k8s pods, VMs, containers) shipping telemetry under
   the same resource attributes, you can add USE/infra panels (CPU,
   memory, restarts, network) with no extra instrumentation — see Phase
   3. Skip if a platform team already owns the host/cluster dashboard.
4. **Which signals are emitted?** traces / logs / metrics. Drives which
   `signoz:signoz_get_field_keys` calls you make.
5. **What's the canonical resource filter?** Usually `service.name`.
   Confirm the exact value before querying — don't guess.
6. **What does success look like for users of this service?** (Sets up
   Phase 2.) Even a one-line answer ("requests under 1s with no 5xx") is
   enough — it becomes the SLI.
7. **Who owns it, and is there a runbook?** Owning team, service tier,
   and the prod/staging scope — these become alert labels and
   annotations in Phase 8. Capture a real runbook URL if one exists;
   never invent one.

Offer "explore data first" as an explicit option.

## Phase 1.5 — Inventory what already exists

Each create sub-skill runs its own rigorous duplicate-check when invoked
(`signoz-creating-dashboards` lists dashboards; `signoz-creating-alerts`
paginates alert rules) — don't re-implement that here. The orchestration
value is a single up-front sweep so you *sequence around* what exists
instead of hitting collisions mid-build:

- **Is the service reporting, and under what exact name?**
  (`signoz:signoz_list_services`.) This name is the resource filter every
  later phase reuses — pin it before anything else.
- **Does a dashboard / alert / saved view already exist for it?** If so,
  plan to **extend** it — route to `signoz-modifying-dashboards`,
  `signoz-managing-views`, or `signoz:signoz_update_alert` — rather than
  standing up a parallel one. `signoz-explaining-dashboards` /
  `signoz-explaining-alerts` summarize anything you're inheriting.
- **Is there a notification channel to reuse?** (Carried into Phase 5.)

Learn just enough to decide extend-vs-create per artifact; leave the
exhaustive listing to the sub-skills.

## Phase 2 — Capture the SLI/SLO before designing anything

This is the gating artifact. Without it, alerts become "the metric
crossed a line" instead of "the user-visible error budget is being burnt
fast enough to matter."

Collect:
- **SLI formula** — `good events / valid events`. E.g. *successful
  checkout requests / total checkout requests* (excluding health checks,
  synthetic probes, scanner traffic, and non-user long-poll endpoints).
- **SLO target** — e.g. 99.5% over a 30-day rolling window.
- **Exclusions** — what doesn't count as a valid event (synthetic
  probes, scanner traffic, streaming endpoints).
- **Error budget remaining** — derived from target. Drives burn-rate
  alert math.

Carry these forward — they feed Phase 8 directly. (SLO/burn-rate
background: `signoz-searching-docs`.)

## Phase 3 — Explore what's actually there (RED + USE coverage)

`signoz-generating-queries` owns the exploration itself — the
discover-before-querying rule, every discovery call
(`signoz:signoz_list_metrics`, `signoz:signoz_get_field_keys`,
`signoz:signoz_get_field_values`,
`signoz:signoz_get_service_top_operations`), tool choice, and the
no-data distinction. Drive it to inventory the data; don't restate its
query mechanics here. (`signoz-searching-docs` covers the RED/USE methods
and SigNoz field semantics if you need a refresher.)

The orchestration job is to map what it finds onto **RED** (Request
rate, Errors, Duration) and **USE** (Utilization, Saturation, Errors),
spot the gaps, and make the judgment calls generating-queries doesn't:

**Coverage checklist** (RED for services, USE for infra/resources):

- [ ] **R**ate — request/call rate per operation
- [ ] **E**rrors — error count or rate per operation
- [ ] **D**uration — p50/p95/p99 latency from spans or histograms
- [ ] **U**tilization — CPU, memory, disk, queue depth, pool occupancy (only if you own the infra)
- [ ] **S**aturation — request queue, GC pause, connection pool wait
- [ ] **E**rrors at the resource layer — disk failures, evictions, OOM kills

A missing signal (e.g. no error count) is an **instrumentation gap** —
flag it before building around what's there.

**Span metrics back alerts, not just panels.** When generating-queries
surfaces SigNoz's trace→metrics output (`signoz_calls_total`,
`signoz_latency_*`, DB/external-call span metrics), prefer it as the RED
source for *both* the dashboard and the burn-rate alerts: it's
pre-aggregated and cheap, whereas raw trace aggregation over a long alert
window is expensive and an unstable alert source