observability-llm-obs

What this skill does

# LLM and Agentic Observability

Answer user questions about monitoring LLMs and agentic components using **data ingested into Elastic** only. Focus on
LLM performance, cost and token utilization, response quality, and call chaining or agentic workflow orchestration. Use
**ES|QL**, Elasticsearch APIs, and (where needed) Kibana APIs. Do not rely on Kibana UI; the skill works without it. A
given deployment typically uses **one or more** ingestion paths (APM/OTLP traces **and/or** integration metrics/logs)—
discover what is available before querying.

## Where to look

- **Trace and metrics data (APM / OTel):** Trace data in Elastic is stored in **`traces*`** when collected by the
  Elastic APM Agent, and in **`traces-generic.otel-default`** (and similar) when collected by OpenTelemetry. Use the
  generic pattern **`traces*`** to find all trace data regardless of source. When the application is instrumented with
  OpenTelemetry (e.g. Elastic
  [Distributions of OpenTelemetry (EDOT)](https://www.elastic.co/docs/solutions/observability/get-started/opentelemetry/use-cases/llms),
  OpenLLMetry, OpenLIT, Langtrace exporting to OTLP), LLM and agent spans land in these trace data streams; metrics may
  land in **`metrics-apm*`** or metrics-generic. Query **`traces*`** and **`metrics*`** data streams for per-request and
  aggregated LLM signals.
- **Integration metrics and logs:** When the user collects data via
  [Elastic LLM integrations](https://www.elastic.co/docs/solutions/observability/applications/llm-observability)
  (OpenAI, Azure OpenAI, Azure AI Foundry, Amazon Bedrock, Bedrock AgentCore, GCP Vertex AI, etc.), metrics and logs go
  to **integration data streams** (e.g. `metrics*`, `logs*` with dataset/namespace per integration). Check which data
  streams exist.
- **Discover first:** Use Elasticsearch to list data streams or indices (e.g. `GET _data_stream`, or
  `GET traces*/_mapping`, `GET metrics*/_mapping`) and optionally sample a document to see which LLM-related fields are
  present. Do not assume both APM and integration data exist.
- **ES|QL:** Use the **elasticsearch-esql** skill for ES|QL syntax, commands, and query patterns when building queries
  against `traces*` or metrics data streams.
- **Alerts and SLOs:** Use the [Observability APIs](https://www.elastic.co/docs/solutions/observability/apis) **SLOs
  API** ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-slo) |
  [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-slo)) and **Alerting API**
  ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-alerting) |
  [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-alerting)) to find SLOs and alerting rules
  that target LLM-related data (e.g. services backed by `traces*`, or integration metrics). Firing alerts or
  violated/degrading SLOs point to potential degraded performance.

## Data available in Elastic

### From traces and metrics (traces*, metrics-apm* / metrics-generic)

Spans from OTel/EDOT (and compatible SDKs) carry **span attributes** that may follow
[OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) or
provider-specific names. In Elasticsearch, attributes typically appear under `span.attributes` (exact key names depend
on ingestion). Common attributes:

| Purpose              | Example attribute names (OTel GenAI)                      |
| -------------------- | --------------------------------------------------------- |
| Operation / provider | `gen_ai.operation.name`, `gen_ai.provider.name`           |
| Model                | `gen_ai.request.model`, `gen_ai.response.model`           |
| Token usage          | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| Request config       | `gen_ai.request.temperature`, `gen_ai.request.max_tokens` |
| Errors               | `error.type`                                              |
| Conversation / agent | `gen_ai.conversation.id`; tool/agent spans as child spans |

Cost is **not** in the OTel spec; some instrumentations add custom attributes (e.g. `llm.response.cost.usd_estimate`).
Discover actual field names from the index mapping or a sample document (e.g. `span.attributes.*` or flattened keys).

Use **duration** and **event.outcome** on spans for latency and success/failure. Use **trace.id**, **span.id**, and
parent/child span relationships to analyze **call chaining** and agentic workflows (e.g. one root span, multiple LLM or
tool-call child spans).

### From LLM integrations

Integrations (OpenAI, Azure OpenAI, Azure AI Foundry, Bedrock, Bedrock AgentCore, Vertex AI, etc.) ship **metrics** (and
where supported **logs**) to Elastic. Metrics typically include token usage, request counts, latency, and—where the
integration supports it—cost-related fields. Logs may include prompt/response or guardrail events. Exact field names and
data streams are defined by each integration package; discover them from the integration docs or from the target data
stream mapping.

## Determine what data is available

1. **List data streams:** `GET _data_stream` and filter for `traces*`, `metrics-apm*` (or `metrics*`), and `metrics-*` /
   `logs-*` that match known LLM integration datasets (e.g. from
   [Elastic LLM observability](https://www.elastic.co/docs/solutions/observability/applications/llm-observability)).
2. **Inspect trace indices:** For `traces*`, run a small search or use mapping to see if spans contain `gen_ai.*` or
   `llm.*` (or similar) attributes. Confirm presence of token, model, and duration fields.
3. **Inspect integration indices:** For metrics/logs data streams, check mapping or one document to see token, cost,
   latency, and model dimensions.
4. **Use one source per use case:** If both APM and integration data exist, prefer one consistent source for a given
   question (e.g. use traces for per-request chain analysis, integration metrics for aggregate token/cost).
5. **Check alerts and SLOs:** Use the SLOs API and Alerting API to list SLOs and alerting rules that target LLM-related
   services or integration metrics, and to get open or recently fired alerts. Firing alerts or SLOs in
   degrading/violated status point to potential degraded performance.

## Use cases and query patterns

### LLM performance (latency, throughput, errors)

- **Traces:** ES|QL on `traces*` filtered by span attributes (e.g. `gen_ai.operation.name` or `gen_ai.provider.name`
  when present). Compute throughput (count per time bucket), latency (e.g. `duration.us` or span duration), and error
  rate (`event.outcome == "failure"`) by model, service, or time.
- **Integrations:** Query integration metrics for request rate, latency, and error metrics by model/dimension as exposed
  by the integration.

### Cost and token utilization

- **Traces:** Aggregate from spans in `traces*`: sum `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` (or
  equivalent attribute names) by time, model, or service. If a cost attribute exists (e.g. custom
  `llm.response.cost.*`), sum it for cost views.
- **Integrations:** Use integration metrics that expose token counts and/or cost; aggregate by time and model.

### Response quality and safety

- **Traces:** Use `event.outcome`, `error.type`, and span attributes (e.g. `gen_ai.response.finish_reasons`) in
  `traces*` to identify failures, timeouts, or content filters. Correlate with prompts/responses if captured in
  attributes (e.g. `gen_ai.input.messages`, `gen_ai.output.messages`) and not redacted.
- **Integrations:** Query integration logs for guardrail blocks, content filter events, or policy violations (e.g.
  [Bedrock Guardrails](https://www.elastic.co/observability-labs/blog/llm-observability-amazon-bedrock-guardrails))
  using the fields defined by that integration.

### Call chaining and agentic workflow orchestration

- **Traces only:** Use **trace hierarchy** in `traces*`. Filter by root service or trace at