kibana-anomaly-detection

What this skill does

# Elastic ML Anomaly Detection

Single skill covering all anomaly detection work against **Kibana Agent Builder** MCP at
`{KIBANA_URL}/api/agent_builder/mcp`. Use the **Mode Selector** below to pick the right approach for the user's question
— modes share the same tool surface and concepts.

## Platform

- Read path: ES|QL against `.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`, `.ml-annotations-*`
- Always-available: `platform.core.execute_esql` (plus additional platform tools for search, index mapping, and
  documentation — see `scripts/agent_builder_constants.json`)
- ML API spec (if available): `.kibana_ai_openapi_spec_elasticsearch` — see
  [references/anomaly-detection-openapi-spec-discover.md](references/anomaly-detection-openapi-spec-discover.md) for
  discovery pattern.
- **Run `ad_validate_ml_tool_permissions` first** when tools return empty/misleading results — missing privileges are
  the most common cause of false negatives. Full permissions matrix:
  [references/permissions-matrix.md](references/permissions-matrix.md).

## Mode Selector

| User intent                                                                   | Mode                                                                                                   |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate**                                                                                        |
| "Why score high/low?" / renormalization / model bounds / forecasts            | **Explain**                                                                                            |
| Missing docs / memory limit / datafeed stopped / CCS / lifecycle / calendars  | **Troubleshoot**                                                                                       |
| Create a job / configure a datafeed / start analysis / retrieve results       | **Manage**                                                                                             |
| Security framing (attack chains, MITRE, exfil)                                | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md)           |
| Observability/SRE framing (degradation, capacity, deployment regression)      | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |

When a question spans modes: **Investigate → Explain → Troubleshoot**. Don't blend mode logic — finish one before moving
on.

---

## Score Quick Reference

- `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational
- `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike)
- `initial_record_score >> record_score` → renormalization (model saw worse anomalies later)
- `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just low value
- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

> Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components:
> [references/score-reference.md](references/score-reference.md).

## Core concepts

Treat `.ml-anomalies-*` as three layers, accessed via `result_type`:

- **`bucket`** — bucket-level unusualness per `bucket_span`. `anomaly_score` is the aggregate across all detectors.
- **`record`** — finest-grained rows with `actual` vs `typical`, `probability`, `record_score`,
  `anomaly_score_explanation`.
- **`influencer`** — entity contributions ranked within a bucket (`influencer_score`).

Read scores this way:

- `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes).
- `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time.
- Compare `actual` to `typical`; use `probability` for raw likelihood.
- Map entities via `partition_field_value` / `by_field_value` / `over_field_value`.
- Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends.

---

## Mode: Investigate — RCA

**When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.

### Tool chain

| Phase                 | Tools                                                                                                          |
| --------------------- | -------------------------------------------------------------------------------------------------------------- |
| Discovery             | `ad_get_available_metadata`, `ad_get_jobs`, `ad_discover_related_jobs`, `ad_discover_jobs_by_datafeed_index`   |
| Timeline / scope      | `ad_query_anomaly_timeline`                                                                                    |
| Cross-job / entities  | `ad_rca_cross_job_entity_match`, `ad_rca_multi_job_entities`, `ad_rca_entity_profile`                          |
| Records / influencers | `ad_query_anomaly_records`, `ad_query_influencers`                                                             |
| RCA depth             | `ad_rca_detector_fingerprint`, `ad_rca_correlation`, `ad_rca_blast_radius`, `ad_rca_score_reassessment`        |
| Evidence / categories | `ad_get_job_datafeed_config`, `ad_rca_source_evidence`, `ad_get_categories`, `ad_search_log_category_examples` |

### Protocol

Follow the 14-step sequence in [references/protocols/investigation.md](references/protocols/investigation.md). High
level: `ad_get_available_metadata` → pair `ad_discover_jobs_by_datafeed_index` with `ad_discover_related_jobs` →
`ad_query_anomaly_timeline` → rank with `ad_rca_multi_job_entities` (`min_job_count=2`) → `ad_rca_detector_fingerprint`
→ drill with `ad_query_anomaly_records` + `ad_query_influencers` (low `min_score=25`) → profile with
`ad_rca_entity_profile` → order with `ad_rca_correlation` → confirm with `ad_rca_source_evidence`. When
`by_field_name == "mlcategory"`, compare with `ad_get_categories` + paired `ad_search_log_category_examples` (baseline
vs. anomaly window).

Finish with a written RCA: **root cause entity · affected jobs · temporal progression · fault class
(resource/network/application) · severity · recommended actions**. Worked example:
[references/worked-example.md](references/worked-example.md). Full ES|QL templates and parameters:
[references/investigate-anomaly-esql-tools.md](references/investigate-anomaly-esql-tools.md).

### Rules

1. **Multi-job entities are prime suspects; single-job entities are usually victims.** Use `min_job_count=2`.
2. **Earliest anomaly timestamp wins** — sort `ad_rca_correlation` by timestamp; first-appearing entity = origin.
3. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes.
4. **Never close an RCA without `ad_rca_source_evidence`** — raw source documents are ground truth.
5. **Use low `min_score` (25 or lower) for influencer queries** — high thresholds miss correlated entities.

---

## Mode: Explain — Score / model behavior

**When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".

### Score types

| Field                  | Scope           | Meaning                                                                 |
| ---------------------- | --------------- | ----------------------------------------------------------------------- |
| `record_score`         | Single record   | Normalized severity after renormalization.                              |
| `initial_record_score` | Single record   | Score at detection time. Gap vs `record_score` = renormalization drift. |
| `anomaly_score`        | Bucket          | Aggregate severity across all detectors in a bucket.                    |
| `influencer_score`     | Entity × bucket |