observability-k8s-investigation

What this skill does

# Kubernetes Investigation

Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the
kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify
root cause across the workload, node, and control-plane layers.

## Scope

**In scope:** OTel-receiver-namespaced indices (`metrics-kubeletstatsreceiver.otel-*`,
`metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*`, `logs-k8sobjectsreceiver.otel-*`) and OTel
semantic conventions (`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.restarts`).

**Out of scope:**

- The legacy Elastic Agent Kubernetes integration (`metrics-kubernetes.*`, `logs-kubernetes.*`, `kubernetes.*` fields).
  Being deprecated — do not author queries against these paths.
- APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain —
  once a K8s root cause is ruled in or out, APM investigation continues outside this skill.
- Cluster provisioning, capacity planning, cost optimization. Different domain.

## Guidelines

These apply to every investigation. When in doubt, re-read them before writing the synthesis.

**Absence of evidence is not evidence. Do not confabulate from empty results.** If log queries return 0 rows, logs are
likely not collected or the pod has no recent lines — this does _not_ mean "dependency unavailable" or any other
specific failure mode. Report `no_logs_available` and weight remaining signals accordingly.

**Empty dependency data ≠ upstream healthy.** Services without APM instrumentation (load generators, workers) emit no
destination metrics. Report `insufficient_dependency_data`, not "upstreams OK."

**Co-symptoms are not causes.** Two services degrading simultaneously usually share an upstream, not a causal link. Only
attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5×
error rate, >3× latency).

**OOMKilled ≠ memory leak by default.** The limit may simply be undersized for the workload's working set. Compare
against a 7-day baseline at the same hour-of-day before claiming a leak.

**Error-termination ≠ application bug by default.** Check `k8s.pod.cpu_limit_utilization` first. CFS throttling driving
liveness probe timeouts is the most common misdiagnosis in this space.

**Average CPU hides throttling.** A pod can look healthy at 40–60% average `cpu_limit_utilization` while being throttled
severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at
max and p95, not just average.

**Restart count is boolean, not a counter.** `k8s.container.restarts` is pulled directly from the K8s API and may be
pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as `== 0` (no recent restarts) vs `> 0`
(recently restarting); do not derive backoff timing or "linear vs exponential" patterns from it. Confirm the restart
pattern via K8s `Killing` / `BackOff` events instead.

**Prefer to report uncertainty over manufacturing confidence.** If the evidence is ambiguous, the synthesis should say
so. Competing hypotheses are a valid output.

## Indices and fields

### Where to look

| Signal                | Index pattern                                       | Use                                                                 |
| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- |
| Pod/container runtime | `metrics-kubeletstatsreceiver.otel-*`               | CPU, memory, network, filesystem. Utilization ratios.               |
| Cluster state         | `metrics-k8sclusterreceiver.otel-*`                 | Restarts, phase, last-terminated reason, HPA, quota, node condition |
| K8s events            | `logs-k8seventsreceiver.otel-*`                     | Killing, BackOff, FailedScheduling, Evicted, image pull events      |
| K8s object snapshots  | `logs-k8sobjectsreceiver.otel-*`                    | Deployment/service/configmap state over time                        |
| Application logs      | `logs-*.otel-*`                                     | `body.text`, `severity_text`, filtered by `k8s.pod.name`            |
| APM                   | `traces-*.otel-*`, `metrics-service_*.otel-default` | Correlate via `service.name` + K8s resource attrs                   |
| ML anomalies          | `.ml-anomalies-*`                                   | Memory-growth, restart-rate, throttle jobs (if configured)          |

### Key fields

Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested `resource.attributes.*` form is for raw
log documents only.

| Field                                            | Index                       | What it is                                              |
| ------------------------------------------------ | --------------------------- | ------------------------------------------------------- |
| `k8s.pod.name`                                   | all k8s                     | Pod name                                                |
| `k8s.namespace.name`                             | all k8s                     | Namespace                                               |
| `k8s.container.name`                             | all k8s                     | Container within pod                                    |
| `k8s.deployment.name`                            | k8sclusterreceiver + others | Parent deployment                                       |
| `k8s.pod.phase`                                  | k8sclusterreceiver          | Pending=1/Running=2/Succeeded=3/Failed=4/Unknown=5      |
| `k8s.container.restarts`                         | k8sclusterreceiver          | Total container restart count                           |
| `k8s.container.status.last_terminated_reason`    | k8sclusterreceiver          | `OOMKilled`, `Error`, `Completed`, `ContainerCannotRun` |
| `k8s.pod.status_reason`                          | k8sclusterreceiver          | Pod-level reason (`Evicted`, `NodeLost`)                |
| `k8s.pod.memory_limit_utilization`               | kubeletstatsreceiver        | 0.0–1.0+ (can exceed 1 transiently before OOM)          |
| `k8s.pod.cpu_limit_utilization`                  | kubeletstatsreceiver        | 0.0–N (frequently >1 under CFS throttling)              |
| `k8s.pod.memory.usage` / `.working_set`          | kubeletstatsreceiver        | Bytes                                                   |
| `k8s.node.condition_memory_pressure`             | k8sclusterreceiver          | 1 = pressure, 0 = ok                                    |
| `k8s.node.condition_ready`                       | k8sclusterreceiver          | 0 = NotReady                                            |
| `k8s.hpa.current_replicas` / `.desired_replicas` | k8sclusterreceiver          | HPA state                                               |
| `attributes.k8s.event.reason`                    | k8seventsreceiver           | Event reason (filter on this)                           |
| `body.text`                                      | k8seventsreceiver / logs    | Event message / log message                             |
| `k8s.object.name`                                | k8seventsreceiver           | involvedObject name (log attribute, use flat form)      |

### Field availability

Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify
presence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis.

| Field                                                        | Why it might be missing                                                                                    | Fall-back