llm-obs-trace-rca

What this skill does

## Backend

**Detection** — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__list_llmobs_evals` appears in your available tools.
3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available.
5. If pup responds → use **pup mode** throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:
   > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs'`) or install pup."

`--backend pup` is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

**pup invocation rules:**
- Invoke via Bash: `pup llm-obs <subcommand> [flags]`
- pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in `[{"type": "text", "text": "<json>"}]`).
- If pup returns an auth error, tell the user to run `pup auth login` and stop.
- Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
- Time flags: pup accepts bare duration strings (`1h`, `7d`, `30m`) and RFC3339 timestamps. Do **not** use `now-`-prefixed strings — strip the prefix when converting from a skill `--timeframe` argument: `now-7d` → `7d`, `now-24h` → `24h`, `now-30d` → `30d`.
- `--summary` on `pup llm-obs spans search` strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.

**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:llm-obs-trace-rca[<inv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:llm-obs-trace-rca:start[<inv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:llm-obs-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode`

# LLM Obs Trace RCA — Root Cause Analysis from Production LLM Traces

Diagnose **why an LLM application is failing** by searching production traces and walking the span tree from symptom to root cause. The skill automatically selects the best analysis mode based on available signals:

| Mode | Signal | When auto-selected |
|------|--------|--------------------|
| **Eval Signal** | LLM judge verdicts and reasoning (pass/fail rates, scoring) | Evaluators are configured for the app |
| **Error Signal** | Runtime errors (`@status:error`, error types, stack traces) | No evals configured, or user explicitly asks about errors/crashes |
| **Generic** | Structural anomalies (latency, agent loops, retrieval misses) | Explicit `mode=generic` override, or no strong signal found in Phase 1 |

The mode is **announced (never asked)** in the first user-facing output with a one-line override hint.

## Methodology

**Resolve → Search → Observe → Open Coding → Axial Coding → Root Cause Navigation → Recommendations**

## Usage

```
What's wrong with <ml_app> over the last <timeframe>
Why is <ml_app> failing today
Analyze eval failures for <eval_name> on <ml_app>
Look at the errors on <ml_app> over the last <timeframe>
Root-cause low scores on <eval_name>
```

## Inputs

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `ml_app` | Yes (or `eval_name`) | — | The application to analyze. |
| `eval_name` | No | — | One or more evaluators to focus on. Implies Eval Signal mode. Pass a list for multi-eval analysis. |
| `timeframe` | No | `now-24h` | How far back to look. |
| `mode` | No | inferred | Explicit mode override: `eval`, `errors`, `generic`. Skips inference entirely. |
| `failure_filter` | No | — | Narrowing scope: `"errors"` (routes to Error Signal path), `"high latency"` (post-fetch duration sort), `"low scores on <eval>"` (promotes to `eval_name`), a tool name or span name (`@name:<x>` query). |

If neither `ml_app` nor `eval_name` is provided, ask the user.

## Available Tools

### Eval discovery & overview

| Tool | Purpose |
|------|---------|
| `list_llmobs_evals` | Discover all configured evals for an `ml_app`. Used in Phase 0 mode inference. |
| `get_llmobs_eval_aggregate_stats` | Pass/fail rate or score distribution for an eval over a time window. |
| `get_llmobs_evaluator` | Full evaluator config: prompt template, assessment criteria, span filter, sampling, provider. Use instead of the deprecated `get_llmobs_eval_config`. |

### Trace & span exploration

| Tool | Purpose |
|------|---------|
| `search_llmobs_spans` | Find spans by tags, span kind, status, query syntax. Paginate with cursor. Entry point for Phase 1. |
| `get_llmobs_span_details` | Metadata, evaluations (scores, labels, reasoning), `status`, error fields, duration, and **`content_info`** map showing available fields + sizes. |
| `get_llmobs_span_content` | Actual content for a span field. Supports JSONPath via `path` param for targeted extraction. |
| `get_llmobs_trace` | Full trace hierarchy as span tree with span counts by kind. |
| `find_llmobs_error_spans` | All error spans in a trace with error type, message, stack, and propagation context. |
| `expand_llmobs_spans` | Load children of collapsed trace nodes. |
| `get_llmobs_agent_loop` | Chronological agent execution timeline (LLM calls, tool invocations, decisions). May return empty — see Phase 4b fallback. |

### Key `get_llmobs_span_content` patterns

| Field | Path | What you get |
|-------|------|-------------|
| `messages` | `$.messages[0]` | System prompt (first message, usually `system` role) |
| `messages` | `$.messages[-1]` | Last assistant response |
| `messages` | *(no path)* | Full conversation including tool calls |
| `input` / `output` | — | Span I/O |
| `documents` | — | Retrieved documents (RAG apps) |
| `metadata` | — | Custom metadata (prompt versions, feature flags, user segments) |

### How to use `search_llmobs_spans`

**Always include `@ml_app:"<ml_app>"` in the `query` string — the structured `ml_app` parameter is unreliable and can return spans from other apps.** Do not rely on the structured parameter alone.

Useful query fragments — combine with space (AND):

| Goal | Query |
|------|-------|
| Errors only | `@status:error` |
| Eval is present on the span | `@evaluations.custom.<eval_name>:*` (presence only — pass/fail is read from `get_llmobs_span_details`, not the query) |
| A specific tool by name | `@name:<tool_name>` |

Dedicated params (`span_kind`, `root_spans_only`, `ml_app`) work alongside `query`, but `query` takes precedence over `tags`.

### Parallelization rules

1. **`get_llmobs_span_details`**: Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a **single message**.
2. **`get_llmobs_span_content`**: Each call is independent — always issue ALL in a single message.
3. **`get_llmobs_trace` / `find_llmobs_error_spans` / `get_llmobs_agent_loop`**: Parallelize across different traces in a single message.
4. **Pipeline parallelism**: Start `get_llmobs_span_details` for page 1 results immediately — don't wait to collect all pages.

---

## Analysis Workflow

**Outpu