signoz-creating-alerts

What this skill does

# Alert Create

Build a SigNoz alert from a user's natural-language intent. The skill targets
two consumers: an autonomous AI SRE agent that runs without a human in the
loop, and a human at a Claude Code / Codex / Cursor prompt. Both go through
the same flow.

## Prerequisites

This skill calls SigNoz MCP server tools (`signoz:signoz_create_alert`,
`signoz:signoz_list_alert_rules`, `signoz:signoz_get_field_keys`, etc.). Before running the
workflow, confirm the `signoz:signoz_*` tools are available. If they are not,
run `signoz-mcp-setup` first to initialize or repair the MCP connection. Do not
try to fall back to raw HTTP calls or fabricate alert configs without the MCP
tools.

## When to use

Use this skill when the user wants to:
- Create, set up, or configure a new alert rule.
- Get paged or notified when a metric, log volume, latency, or error rate
  crosses a threshold.
- Detect anomalous behavior on a service, host, or signal.
- Catch silent data loss ("alert if data stops arriving from X").

Do NOT use when the user wants to:
- Understand what an existing alert monitors → `signoz-explaining-alerts`.
- Diagnose why an existing alert fired → `signoz-investigating-alerts`.
- Modify thresholds, queries, or routing on an existing alert → call
  `signoz:signoz_update_alert` directly.

## Required inputs (strict)

Alert creation is a write operation against a shared system. Guessing here
creates noisy alerts on the wrong service that someone else has to clean up.
The skill enforces a strict input contract:

| Input | Required | Source if missing |
|---|---|---|
| Alert intent (NL goal) | yes | `$ARGUMENTS` or recent user turn |
| Resource attribute filter (e.g. `service.name`, `k8s.namespace.name`, `host.name`) | yes | discover via `signoz:signoz_get_field_keys` + `signoz:signoz_get_field_values` |
| Threshold value(s) | inferred from intent | derive a sensible default and surface in the preview |
| Severity | inferred from intent | default `warning`; promote to `critical` only if user said "page", "wake up", "critical" |
| Notification channel | yes | `signoz:signoz_list_notification_channels` + offer "create new" |

If a required input is missing and cannot be discovered, **stop before
calling any write tool** and ask the user. The host application decides
how the question is surfaced (a structured clarification tool, inline
`<assistant_question>` tags, an interactive prompt, etc.) — follow the
host's UI rendering rules.

What to include in the question:

- **What is missing** — name the input concretely (e.g. "which
  resource-attribute filter to use").
- **Candidate lists** populated from your discovery calls — concrete
  values per attribute the user can pick from. Example shape:
  `service.name` → `frontend`, `checkout`, `payments`;
  `host.name` → `prod-api-1`, `prod-db-1`.
- **Allow free-form input** so the user can name a value you didn't
  surface.

In autonomous mode (no human), escalate to the caller or fill the gap
from upstream context. Either way, do not proceed to
`signoz:signoz_create_alert` with a guessed value.

## Workflow

### Step 1: Parse intent and check what's missing

Extract from the user's request:
1. **What to monitor** — signal type (metrics / logs / traces / exceptions)
   and the specific condition (CPU, error rate, p99 latency, log count, ...).
2. **Resource scope** — which service, host, namespace, or environment.
3. **Threshold** — numeric value and comparison ("above 80%", "below 100/s").
4. **Severity** — implicit from urgency words ("page" → critical, default
   warning otherwise).
5. **Channel** — explicit channel name if the user provided one.

Map signal phrasing to alert type:

| User says | alertType | signal |
|---|---|---|
| metric, CPU, memory, latency, request rate | METRIC_BASED_ALERT | metrics |
| log, error logs, log volume, log pattern | LOGS_BASED_ALERT | logs |
| trace, span, latency p99, slow requests | TRACES_BASED_ALERT | traces |
| exception, stack trace, crash | EXCEPTIONS_BASED_ALERT | (clickhouse_sql) |

If resource scope is missing, run discovery (Step 2). If still missing after
discovery, stop and ask the user (see *Required inputs* above).

### Step 2: Discover resource attributes and metric names

When the user does not name a service / host / namespace, the SigNoz MCP
guideline applies: **always prefer a resource-attribute filter**. Discover
candidates instead of guessing:

1. Call `signoz:signoz_get_field_keys` with `fieldContext=resource` to enumerate
   resource attributes for the chosen signal.
2. Call `signoz:signoz_get_field_values` for the most likely attribute (typically
   `service.name`, then `host.name`, then `k8s.namespace.name`) to get
   concrete values.
3. If the user mentioned a metric by name, call `signoz:signoz_list_metrics` with a
   search term to verify the exact OTel metric name. Wrong names create
   alerts that never fire.

Surface the candidates in your clarification request (see *Required
inputs* above). Do not pick one.

### Step 3: Check for duplicate alerts

Once the scope is resolved (either provided by the user or discovered in
Step 2), check for existing alerts before probing data or authoring a
new config — both are wasted work if the user wants to update an
existing rule instead.

Call `signoz:signoz_list_alert_rules` and **paginate through every page** —
`pagination.hasMore` is true until you have walked the full list. This lists
*configured* alert rules (the durable state); do not use `signoz:signoz_list_alerts`,
which returns currently triggered/active alert instances and will silently
miss rules that are configured but not firing right now. Check for existing
rules that match the user's intent (same signal + same scope + similar
threshold). If a likely duplicate exists, surface it and ask whether to
create a new one anyway, modify the existing one (out of scope here — use
`signoz:signoz_update_alert`), or cancel.

### Step 4: Probe data existence for the chosen filter (fail fast)

Before authoring any alert config, confirm the **specific combination** the
alert will watch (metric × service × any other filter) actually emits data.
The most common silent failure is "metric exists in the catalog *and* the
service exists in the catalog, but the service doesn't emit that metric"
— each piece checks out in isolation, the alert saves successfully, and it
silently never fires.

Run a single probe over the last 1 hour using the same filter the alert
will use, but with the simplest aggregation that confirms data exists:

- **Metrics**: `signoz:signoz_execute_builder_query` with `count()`
  (or `count_distinct(service.name)` if scope-discovering). Use
  `signoz:signoz_query_metrics` when you already have a concrete
  `metricName` — it auto-applies aggregation defaults and accepts
  `filter`/`groupBy`, but requires a concrete `metricName` (no PromQL,
  no filter-only probes).
- **Logs**: `signoz:signoz_aggregate_logs` with `count()` over the filter.
- **Traces**: `signoz:signoz_aggregate_traces` with `count()` over the filter.

Inspect the result:

- **Probe returns rows** → proceed to Step 5.
- **Probe returns empty** → STOP. Do not build an alert config the user
  will then be asked to throw away. Stop and ask the user (see *Required
  inputs* above), describing what was missing and offering concrete
  recovery:
  - Service doesn't emit the metric → call
    `signoz:signoz_get_field_values signal=metrics name=service.name metricName=<metric>`
    to list the services that *do* emit it; let the user pick a different
    service or a different metric.
  - Wrong attribute name (`service` instead of `service.name`) → suggest
    the semantic-convention name and re-probe.
  - Service emits the metric but not in the expected time range → widen
    the probe window once (e.g. last 24h) before declaring no-data.

**Exception — log-based crash / panic / OOMKilled / FATAL alerts.** These
intentionally have zero matches in a healthy system. The probe will
return emp