exploring-signals-scouts

What this skill does

# Exploring Signals scouts

A **scout** is a scheduled agent that wakes on its own interval, looks at one PostHog project,
decides what's genuinely worth surfacing, and either emits it as a **finding** into the Signals
inbox or closes out empty (a real, valid outcome). PostHog ships a fleet of canonical scouts — a
cross-product generalist (`signals-scout-general`) plus per-surface specialists
(`-error-tracking`, `-ai-observability`, `-logs`, `-revenue-analytics`, `-surveys`,
`-csp-violations`, `-observability-gaps`). A project may also have **custom scouts** beyond the
canonical fleet — any `signals-scout-*` skill a team authored (e.g. `-brand-mentions`,
`-mcp-feedback`) shows up here too, so don't assume the roster is only the canonical set.

This skill helps you **understand and explore what a project's scouts are doing and how they're
performing** — entirely through read-only MCP tools. It is the observability counterpart to
[`authoring-signals-scouts`](../authoring-signals-scouts/SKILL.md) (which teaches writing and
tuning) and to [`inbox-exploration`](../inbox-exploration/SKILL.md) (which covers the inbox
reports scouts feed into).

There are five things you can observe about the fleet, each with its own tool:

| What you want to know                        | Tool                                     | What it tells you                                                               |
| -------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------- |
| Which scouts run, how often, in what posture | `signals-scout-config-list`              | One row per scout: schedule, `enabled`, `emit`, `last_run_at`                   |
| What the scouts actually did, run by run     | `signals-scout-runs-list` / `-retrieve`  | Per-run status, timing, end-of-run summary, `emitted_count`, deep-link          |
| What the fleet has learned across runs       | `signals-scout-scratchpad-search`        | Durable per-team memory (baselines, noise, allowlists)                          |
| What the scouts actually **emitted**         | `execute-sql` over `document_embeddings` | The authoritative per-finding record (weight, severity, confidence) — see below |
| What the scouts surfaced to the user         | `inbox-reports-list`                     | Findings that cleared the bar and became inbox reports                          |

The orienting sixth is `signals-scout-project-profile-get` — the deterministic snapshot of "what's
true about this project" that every scout cold-starts from. When a scout found nothing, this is
usually why.

## Output handling: expect to offload to a file

Two of these tools — `signals-scout-runs-list` and especially
`tasks-runs-session-logs-retrieve` — routinely return payloads that **overflow an MCP client's
token budget and get spilled to a file**. This is the normal path, not an error. Plan for it up
front rather than discovering it after a failed call:

- **Keep `limit` small** on `signals-scout-runs-list` (~10–15). Each row carries a long prose
  `summary`, and runs come back newest-first across the _whole_ fleet, so even a modest page is
  large.
- **Session logs are large by nature.** A single run's log is hundreds of KB to a few MB. Fetch it
  with **`call --json`** (so the saved file is real JSON, not the pretty text format — `jq`-able)
  and read the saved file with `jq` / a script rather than inline.
- **Don't hand-parse the session log.** The bundled [`scripts/`](#helper-scripts) do the
  reconstruction for you — see below.

## Start here: is the fleet even set up?

Don't assume the project has scouts. The fleet only runs on teams enrolled via the `signals-scout`
feature flag, and a project may have no configs, all-disabled scouts, or scouts stuck in dry-run.
Run this first whenever a user asks about their scouts for the first time in a session.

```json
signals-scout-config-list
```

Read the result against three cases:

The config list is unpaginated — it comes back as `{ results: [...] }` (a bare array), with no
`count` field. Read the result against three cases:

- **Empty (`results: []`)** — no scouts are registered. The project isn't enrolled in the scout
  fleet (or hasn't ticked yet). Say so plainly; don't go fishing for runs. Point the user at the
  Signals scout settings / PostHog Code onboarding rather than inventing activity.
- **Configs exist but all `enabled: false`** — the fleet is registered but paused. Nothing is
  running. Tell the user which scouts exist and that they're all off.
- **At least one `enabled: true`** — the fleet is registered and that scout is allowed to run. For
  each enabled scout note its `run_interval_minutes` (cadence), `emit` (false = **dry-run**, runs
  but writes nothing to the inbox), and `last_run_at`. One caveat before reporting "it's live": runs
  are gated by the `signals-scout` feature flag, not by `enabled`. A project that was enrolled and
  later drained from the flag keeps its `enabled: true` rows, but the coordinator no longer plans
  runs for it — so a stale or `null` `last_run_at` on an enabled scout usually means the project is
  no longer enrolled, not that the scout is idle.

  **`last_run_at` is a _dispatch_ stamp, not proof a run executed.** The coordinator advances it the
  moment it _enqueues_ a child workflow for a due scout — before any worker picks the run up. Child
  dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no
  run ever materializes, yet `last_run_at` keeps marching forward each tick. So a recent
  `last_run_at` means "dispatched this tick," **not** "a run is genuinely happening." The
  authoritative liveness signal is the newest actual **run row** in `signals-scout-runs-list`, not
  the config stamp. Cross-check them: if `last_run_at` is fresh (minutes ago) but no run row has
  appeared for that scout in well over its `run_interval_minutes`, the fleet is **dispatching but
  not running** — workers backed up / down, or runs stranded — a real reliability problem, not a
  live scout. Don't report "it's running" off `last_run_at` alone.

A scout that is `enabled: true` but `emit: false` is the most common source of "my scout isn't
doing anything" confusion: it _is_ running and reasoning every tick, it just isn't allowed to post
findings yet. Always surface the `emit` posture when reporting on a scout.

See [`references/scout-data-model.md`](references/scout-data-model.md) for every field on a config,
run, and scratchpad entry, the run status values, and how the pieces link together.

## Workflow: survey the fleet

"What scouts do I have / what are they doing?" — lead with `config-list`, then enrich with the
most recent run per scout so the user sees liveness, not just configuration.

1. `signals-scout-config-list` — the roster.
2. For each enabled scout, `signals-scout-runs-list` and pick the newest run with a matching
   `skill_name` (runs come back newest-first across the whole fleet, so a single call usually
   covers everyone). Report `status` and how long ago it ran.

Present it as a table the user can scan — scout, cadence, posture, last run, last outcome — and
call out anything anomalous (never run, last run errored, stuck in dry-run for a long time).

## Workflow: understand one scout end to end

"How does my error-tracking scout work / how is it doing?"

1. **Read its config** — find the row in `config-list` for `signals-scout-error-tracking`:
   schedule, posture, last run.
2. **Read its body** — `posthog:llma-skill-get {"skill_name": "signals-scout-error-tracking"}`
   returns the team's actual instruction set (which may be a canonical default or a diverged,
   hand-edited row). This is what the agent is told to do every run — its signal-vs-noise
   discriminator, explore patterns, and disqualifiers. To understand _why_ a scout behaves the
   way it does, read its body.
3. **Read its recent runs** — `ru