llm-obs-session-classify

What this skill does

## Backend

**Detection** — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__search_llmobs_spans` appears in your available tools.
3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available.
5. If pup responds → use **pup mode** throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:
   > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs,rum'`) or install pup."

`--backend pup` is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

**pup invocation rules:**
- Invoke via Bash: `pup llm-obs <subcommand> [flags]`
- pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in `[{"type": "text", "text": "<json>"}]`).
- If pup returns an auth error, tell the user to run `pup auth login` and stop.
- Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
- Time flags: pup accepts bare duration strings (`1h`, `7d`, `30m`) and RFC3339 timestamps. Do **not** use `now-`-prefixed strings — strip the prefix when converting from a skill `--timeframe` argument: `now-7d` → `7d`, `now-24h` → `24h`, `now-30d` → `30d`.
- `--summary` on `pup llm-obs spans search` strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

**pup mode notes by entry mode:**
- `session_id` mode: Steps 1–3 and Step 5 work fully. Step 4 (RUM) uses `pup rum aggregate --user-email EMAIL` instead of `analyze_rum_events` — see Tool Reference. Step 4b (audit trail) is pup-native and queries the active user's own org via OAuth.
- `trace_id` mode: Full parity with MCP mode.
- `ml_app` mode: Option A (`aggregate_spans`) is unavailable in pup — skip it and proceed directly to Option B.

**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.

**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:llm-obs-session-classify[<inv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:llm-obs-session-classify:start[<inv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:llm-obs-session-classify:start[3a9f1c2b] — Step 1: enumerate turn root spans for session abc-123`

# Skill: eval-session-classify

Classification skill for Datadog LLM Obs sessions and traces. Produces a satisfaction verdict (`yes` / `partial` / `no`) with a brief reasoning string. Designed to be called standalone or by `eval-pipeline`.

---

## Inputs

| Input | Mode | Required | Description |
|-------|------|----------|-------------|
| `session_id` | session_id mode | Yes | UUID of a Datadog CMD+I assistant session |
| `trace_id` | trace_id mode | Yes | Trace ID from LLM Observability |
| `ml_app` | ml_app mode | Yes | LLM app name to sample from |
| `timeframe` | ml_app mode | No (default: `now-7d`) | How far back to sample |
| `sample_limit` | ml_app mode | No (default: `20`, cap: `50`) | Number of sessions or traces to classify |

**If none of `session_id`, `trace_id`, or `ml_app` is provided → stop immediately and return:**

```json
{
  "error": "missing_input",
  "detail": "Provide one of: session_id, trace_id, or ml_app."
}
```

---

## Phase 0 — Mode Detection

- `session_id` provided → **session_id mode** → go to [Session Mode](#session-mode)
- `trace_id` provided (no `session_id`) → **trace_id mode** → go to [Trace Mode](#trace-mode)
- `ml_app` provided (no `session_id`, no `trace_id`) → **ml_app mode** → go to [ML App Mode](#ml-app-mode)
- Nothing provided → emit error (see above)

---

## Output Format

**Default (compact) — used in all modes unless the caller requests verbose:**

```
verdict: yes | partial | no
reason: <one sentence>
```

**Verbose** — full markdown report (see end of each mode section). Request verbose by including `verbose: true` in input or asking for a detailed report.

In ml_app mode, a summary table is always appended after the per-trace compact blocks.

---

## Content Retrieval Cascade

Reading conversation content follows this cascade across all modes. Run the cascade for every turn root span identified in Step 1 (session mode) or Step T1 (trace mode) before forming a verdict.

> **Completeness gate**: do not proceed to RUM (session mode) or classification (any mode) until the cascade has run for every turn. Previews from `search_llmobs_spans` (`input.preview` / `output.preview`, truncated to ~200 chars) do NOT satisfy this gate — they identify turn boundaries, nothing more. Only actual content from `get_llmobs_agent_loop`, `expand_llmobs_spans` + `get_llmobs_span_content`, or `get_llmobs_span_content(field="input"/"output")` counts.
>
> **Short sessions** (≤ 20 turns): run C1/C2 for every turn — no selection, no "key turns" heuristic.
>
> **Long sessions** (> 20 turns): run C1/C2 for the **first 5** and **last 5** turns, plus any turn whose `input.preview` or `output.preview` flags an anomaly (error, correction loop, repeated intent, unexpected tool). Scan all remaining turns' previews for anomalies before skipping them.

**Parallelism**: issue all N `get_llmobs_agent_loop` calls in a single message — one call per turn, all in the same batch. For an N-turn session this means N simultaneous C1 calls. Do not process turns sequentially and do not form partial verdicts before all results are in hand. Move to C2 for a turn only when C1 returns `iterations: []` for that turn.

### C1 — `get_llmobs_agent_loop(trace_id, agent_span_id)`

The richest source: full system prompt, user message, tool call arguments + results, assistant response, and token economics per iteration. Attempt this first for every agent span.

```
get_llmobs_agent_loop(
  trace_id           = "<TRACE_ID>",
  span_id            = "<AGENT_SPAN_ID>",
  from               = "now-90d",
  to                 = "now",
  max_content_length = 2000
)
```

- **`iterations: []` and `timeline: null`** → the app's LLM spans go through an intermediate workflow layer (e.g. `get_answer_from_model_step`) rather than as direct children of the agent span. Fall through to C2.
- **404** → span ID does not resolve in the trace store. Fall through to C2.
- **Content fields contain `<REDACTED_INPUT>` or `<MASKED_STREAMING_RESPONSE>`** → IO tracing is disabled by a feature flag. Structure (iteration count, tool names, token usage) is still useful — note it. Fall through to C2 for actual content.

**When a C1 result is too large for context and gets persisted to disk** (common on long sessions): do NOT try to Read the full file — it will exceed the token limit. Run a targeted Python extraction to build a structured per-turn summary:

```python
import json
with open('<persisted_path>') as f:
    data = json.loads(json.load(f)[0]['text'])
iters = data['iterations']
for it in iters:
    tcs = [tc['name'] for tc in it.get('tool_calls', [])]
    content = it.get('content', '') or ''
    # Skip the system prompt body — it's verbose and not what we're classifying.
    if len(content) > 4000 and content.