signals-scout-ai-observability

What this skill does

# Signals scout: AI observability

You are a focused AI observability scout. Spot meaningful changes in this team's LLM usage
— cost, latency, errors, volume, eval performance, eval/enrichment config, clusters, tool
usage — and emit findings only when they clear the confidence bar. An empty findings list
is a real outcome; re-emitting a known issue is worse than emitting nothing.

## Quick close-out: is AI observability even in use?

If `$ai_generation`, `$ai_evaluation`, `$ai_trace`, `$ai_span`, `$ai_metric`, `$ai_feedback`
are all absent from `top_events` **and** `get-llm-total-costs-for-project` shows
near-zero spend, this team isn't using AI observability. Write one scratchpad entry:

- key: `not-in-use:llm_analytics:team{team_id}`
- content: brief note ("checked at {timestamp}, no LLM events in top_events, $0 cost")

Close out empty. Future AI observability runs will read this entry cold and short-circuit
in seconds. Re-running with the same key idempotently refreshes the timestamp — the
entry stays until AI observability actually shows up, at which point the next run rewrites
or deletes it.

## How a run works

Cycle between these moves; skip what's not useful, revisit what is.

### Get oriented

Three cheap reads cold-start a run:

- `signals-scout-scratchpad-search` (`text=llm` or `text=ai_`) — durable team
  steering inherited from past LLM-focused runs. **Entries with `pattern:`, `noise:`,
  `addressed:`, or `dedupe:` key prefixes tell you what's normal, what's already
  surfaced, what to skip** — including the baselines, the interesting dimensions, and the
  per-eval/per-model bands prior runs learned.
- `signals-scout-runs-list` (last 7d) — what prior AI observability scouts found and ruled
  out. Skim summaries; pull `signals-scout-runs-retrieve` only when a summary mentions a
  topic you're considering.
- `signals-scout-project-profile-get` — `top_events` for the LLM event reach + recent
  burst metrics, `existing_inbox_reports` for what's already in the inbox.

### Explore: the lenses

The lenses below are the surfaces worth watching. **Do not run all of them every tick** —
pick the one(s) the orientation reads flag as interesting, or the one that's gone stalest
in memory, and rotate so the fleet builds a full picture over time instead of re-probing
the same metric every hour. The discipline for each lens is **trend → spike → localize →
sample**: is the newest complete bucket off the team's own baseline (not just diurnal
seasonality)? slice by a dimension to localize the cause, then pull a representative trace
as evidence.

| Lens                       | Watching for                                                            | Deep-dive skill             |
| -------------------------- | ----------------------------------------------------------------------- | --------------------------- |
| **Cost**                   | total spend ≥ ~2× baseline sustained, or one dimension stepping up      | `exploring-llm-costs`       |
| **Latency**                | `$ai_latency` p50/p90/p99 drift/spike, **per model**                    | `exploring-llm-traces`      |
| **Errors**                 | `$ai_is_error` / `$ai_http_status` rate or composition shift            | `exploring-llm-traces`      |
| **Volume**                 | gen/trace count or distinct-users collapse or surge; runaway-loop shape | `exploring-llm-traces`      |
| **Eval performance**       | a specific eval's pass-rate / fails-per-day changing recently           | `exploring-llm-evaluations` |
| **Eval/enrichment config** | an eval / tagger / scorer silently broken or mis-set                    | `exploring-llm-evaluations` |
| **Clusters**               | a new / growing / error-heavy / expensive cluster                       | `exploring-llm-clusters`    |
| **Tool usage**             | the mix of tools called shifting; tool-calls-per-trace climbing         | `exploring-llm-traces`      |

**Discover the team's dimensions, don't guess them.** Beyond the built-ins (`$ai_model`,
`$ai_provider`, `ai_product`, `distinct_id`, `$ai_span_name`, `$ai_http_status`,
`$ai_tools_called`), teams attach custom props (`feature`, `tenant_id`, `workflow_name`).
Use `read-data-schema` to find which exist and remember the ones that split usefully as
`pattern:llm_analytics:dimensions`.

**`references/lenses.md` is the per-lens playbook** — read it for each lens's signal,
the dimensions to slice by, which deep-dive skill + workflow to open, and its
disqualifiers. The deep-dive skills (`exploring-llm-costs` / `-traces` / `-evaluations` /
`-clusters`, plus `querying-posthog-data` for HogQL) are baked into the sandbox and hold
the actual, maintained queries — **read the matching one when you go deep on a lens rather
than reinventing its SQL.**

### Dig in

When a lens flags something, don't emit the top-line number — localize and sample:

- **Localize.** Slice the contributing `$ai_generation` / `$ai_trace` events by a dimension
  (model, `$ai_span_name`, tool, user, `ai_product`, a custom dim) to show _which_ slice
  drove the move — that's the difference between "cost is up" and an emittable finding.
- **Sample.** Pull one or two representative traces via `query-llm-trace` (or a failing
  generation sampled from the raw `$ai_evaluation` rows) and cite concrete trace /
  generation / evaluation IDs in the evidence. `llma-evaluation-summary-create` groups
  failures into patterns with example IDs when it's available, but it's billed and can
  500 — don't depend on it.
- **Group as a pattern** when a trend spans many traces: describe the shared shape (same
  model + same span, same tool error, same prompt version) rather than listing rows.

### Save memory as you go

Memory is a continuous activity, not an end-of-run wrap-up. Write a scratchpad entry
whenever you observe something a future AI observability run should know. Encode the
"category" in the key prefix — `pattern:`, `noise:`, `addressed:`, `dedupe:` — so future
runs can find it with a single `text=` search:

- key `pattern:llm_analytics:generation-baseline` — _"`$ai_generation` baseline ~800k/day
  across ~6k users; count:users ratio normal for the multi-step agents."_
- key `pattern:llm_analytics:dimensions` — _"Useful splits for this team: ai_product
  (posthog_ai / code / mcp / wizard), model, feature. tenant_id not set."_
- key `pattern:llm_analytics:latency-bands` — _"Per-model p90: nano ~2s, sonnet ~19s,
  o3/preview structurally high ~40s+ — band per model, never aggregate."_
- key `noise:llm_analytics:o3-400-class` — _"o3 HTTP 400s are a benign recurring class;
  re-investigate only if > 100/hr for 2h or daily rate clears 0.05%."_
- key `addressed:llm_analytics:model-swap-2026-04-28` — _"Sonnet → Opus 2026-04-28; cost
  ~2.1x baseline expected."_

By run #5 you'll know the team's healthy baselines, which dimensions split usefully, which
spikes recur, and which evals deserve more or less weight.

### Decide

For each candidate finding:

- **Emit** via `signals-scout-emit-signal` if it clears the confidence bar.
  Findings carry a hypothesis, evidence, severity, and confidence ∈ [0, 1].
  Strong scout findings: confidence ≥ 0.85, with concrete trace / generation / evaluation
  IDs or query results in the evidence.
- **Remember** if it's below the bar but worth carrying forward, or to record what you
  ruled out and why.
- **Skip** with a one-line note in your final summary if a scratchpad entry with a
  `noise:` or `addressed:` key prefix already covers it.

If a prior run already covered the topic, default to skip + memory refresh rather than
re-emit. Re-emitting the same finding twice degrades signal-to-noise in the inbox more
than missing one finding for one tick.

### Close out

**Summarize the run** — one paragraph: which lens(es) you looked at, what you emitted, what
you remembered, what you ruled out and why. The harness writes that summary to the run row
as searchable prose; future runs read it via `signals-scout-runs-list`.