investigating-with-observability

What this skill does

# Troubleshooting with Observability Skills

## Core Principle

Random querying wastes time and produces misleading results. Empty results from wrong metric names look identical to "no problem exists." Jumping between signals without a hypothesis leads to thrashing.

**Discover before you query. Hypothesize before you correlate. Confirm before you conclude.**

If you haven't completed Phase 1, you cannot propose root causes. If you haven't correlated across at least two signal types, your conclusion is a guess.

## The Investigation Protocol

Complete each phase before proceeding to the next.

```
Phase 1: Gather Signals     → What's already known? What's alerting?
Phase 2: Discover and Scope  → What data exists? What are the real names?
Phase 3: Hypothesize and Test → Form one theory, query to confirm or refute
Phase 4: Correlate and Confirm → Cross-reference across signal types, find root cause
```

### Phase 1: Gather Signals

Before writing any query, establish what's already known.

**1. Check env var availability** — run the gating check from the Subagent Dispatch section.

**2. Dispatch signal-gathering subagents in parallel:**

| Subagent | Condition | What it does |
|----------|-----------|-------------|
| AlertManager check | `VM_ALERTMANAGER_URL` available | Checks VM alerts + AlertManager alerts and silences |
| Metrics discovery (alerts only) | `VM_METRICS_URL` available AND `VM_ALERTMANAGER_URL` NOT available | Checks VM alerts as fallback when AlertManager agent can't be dispatched |

If `VM_ALERTMANAGER_URL` IS available, the AlertManager check agent handles BOTH VM alerts and AlertManager queries — no need to dispatch a separate metrics agent for alerts.

Read the agent prompt files and dispatch in a single Agent tool call. Include in each subagent's prompt:

- The agent file content
- The investigation target (namespace, service, or component name if known)

**3. Synthesize results** — once subagents return:

- Combine alert findings from all sources
- Establish a timeline: when did symptoms start? What changed?
- If timeline is unclear, ask the user

**4. Identify which signal type to start with:**

| Symptom | Start with | Then correlate with |
|---------|-----------|-------------------|
| Resource/rate issue | Metrics | Logs |
| Errors/crashes | Logs | Metrics |
| Latency/slow requests | Traces | Logs |
| Alert firing | Metrics (alert details) | Logs + Traces |

### Phase 2: Discover and Scope

**Never guess metric names, log field names, or service names.** Discovery is not optional — it prevents the single most common investigation failure: drawing conclusions from empty results caused by wrong names.

**Dispatch discovery subagents in parallel** for ALL available backends. Read each agent prompt file and dispatch in a single Agent tool call. Include in each subagent's prompt:

- The agent file content
- Target namespace and/or service name (from Phase 1 findings)
- Time range for the investigation (RFC3339 format)
- Any specific keywords or components to search for

| Subagent | Condition |
|----------|-----------|
| Metrics discovery | `VM_METRICS_URL` available |
| Logs discovery | `VM_LOGS_URL` available |
| Traces discovery | `VM_TRACES_URL` available |

**Synthesize discovery results:**

- Merge discovered names across all backends
- Note which backends have data for the target and which don't
- Identify the richest signal source for Phase 3 hypothesis testing

**Consult skill references for complex queries.** You do NOT know LogsQL syntax from training data — it is NOT Loki LogQL. For complex queries beyond what the subagents already ran, invoke the corresponding `*-query` skill or use the LogsQL Quick Reference below.

### Phase 3: Hypothesize and Test

After discovery, form a specific hypothesis before querying further.

**State it clearly:** "I think [component X] is [failing/slow/OOM] because [evidence Y from Phase 1]."

**Test minimally:**

- Query ONE thing to confirm or refute the hypothesis
- Don't query everything at once — you'll drown in data
- Use instant queries first (cheaper, faster) before range queries

**If the hypothesis is wrong:**

- Don't add more queries on top — form a NEW hypothesis
- Re-examine what Phase 1 and Phase 2 revealed
- Ask: did discovery show anything unexpected?

**After 3 failed hypotheses: STOP.**
Three wrong guesses means you're missing something fundamental. Either:

- A key data source hasn't been discovered yet
- The scope is wrong (different namespace, different service, different time range)
- You need to ask the user for more context

### Phase 4: Correlate and Confirm

A single signal type is not proof. Correlate across at least two before concluding.

**Dispatch correlation subagents in parallel** for the signal types you need. Reuse the same agent prompt files from `agents/`, but provide specific queries rather than discovery tasks. Include in each subagent's prompt:

- The agent file content
- The specific query to run (metric expression, log filter, trace search parameters)
- The exact time range to query (narrowed from Phase 3 findings)
- What to look for (the confirmed hypothesis from Phase 3)

Example parallel dispatch for correlation:

- Metrics agent: "Query `rate(http_requests_total{code=~'5..', namespace='myapp'}[5m])` from T1 to T2"
- Logs agent: "Search `{namespace='myapp'} error` from T1 to T2, return sample messages"
- Traces agent: "Search traces for service `myapp` with `minDuration=1s` from T1 to T2"

**Correlation techniques:**

- **Time-based**: Identify anomaly timestamp in metrics, query logs/traces at that time
- **Trace ID**: Find trace IDs in traces, search logs for `trace_id:"<id>"`
- **Pod name**: Get pod name from metrics labels, use it in log stream filters

**Only after correlation:** propose root cause and remediation.

## Red Flags — STOP and Return to Phase 1

If you catch yourself:

- Proposing a root cause after querying only one signal type
- Writing a LogsQL query from memory without checking syntax
- Querying a metric name you haven't confirmed exists via discovery
- Getting empty results and concluding "no problem"
- Skipping the alerts check because "it's probably not that"
- Running five different queries hoping one shows something
- Saying "let me just try..." instead of forming a hypothesis

**All of these mean: STOP. You're guessing, not investigating.**

## Common Rationalizations

| Excuse | Reality |
|--------|---------|
| "I know the metric name" | Maybe. Discovery takes 2 seconds and prevents 20 minutes of chasing empty results. |
| "Alerts won't help here" | Alerts are free to check and frequently contain the exact answer. Skip at your peril. |
| "Just need to check logs quickly" | Quick log checks without discovery produce wrong field names and misleading results. |
| "Empty results = no problem" | Empty results more often mean wrong query than absent problem. Verify names first. |
| "I'll correlate later" | Single-signal conclusions are guesses. Correlate before claiming root cause. |
| "LogsQL is like LogQL/Elasticsearch" | It's not. The syntax differences cause silent failures. Consult the reference. |

## Environment

Environment is controlled by env vars. Check current state:

```bash
echo "VM_METRICS_URL:      $VM_METRICS_URL"
echo "VM_LOGS_URL:         $VM_LOGS_URL"
echo "VM_TRACES_URL:       $VM_TRACES_URL"
echo "VM_ALERTMANAGER_URL: $VM_ALERTMANAGER_URL"
if [ -n "${VM_AUTH_HEADER-}" ]; then
  echo "VM_AUTH_HEADER:      (set)"
else
  echo "VM_AUTH_HEADER:      (empty - no auth)"
fi
```

If unsure which environment the application runs in, ask user.

## Subagent Dispatch

This skill dispatches parallel subagents at phase boundaries to speed up investigations. Each subagent carries embedded API reference and returns structured findings.

### Env Var Gating

Before each dispatch round, check which backends are available:

```bash
echo "METRICS:${VM_METRICS_URL:+available}"
echo "LOGS:${VM_LOGS_URL:+available}"
echo