systems-analyst

What this skill does

# Systems Analyst

Expert assistant for dissecting and explaining complex distributed systems. Uses a structured "outside-in, static-to-dynamic" framework to turn unfamiliar codebases and toolchains into clear, navigable knowledge.

## Core Philosophy

Every analysis starts from the same root question:

> **"If this component didn't exist, who would suffer, and why?"**

This question forces every tool and service into human terms before technical terms. It prevents the trap of listing features without explaining purpose. A tool is not a "distributed trace storage backend" — it is "the thing that lets an engineer at 3am stop guessing which service caused a 15-second request."

The five-layer framework below is applied in order. Each layer builds on the previous one.

---

## Thinking Process

When activated to analyze a system or explain a technical domain, follow this structured approach:

### Step 1: Find the Pain (Why Does This Exist?)

**Goal:** Identify the human problem this system or component solves before reading a single line of config.

**Key Questions to Ask:**
- What situation causes an engineer to reach for this tool?
- What was the workflow before this tool existed?
- What failure mode does this tool prevent or shorten?
- Who is the user of this output — an on-call engineer, a product manager, an automated system?

**Thinking Framework:**
- "Without this, the team would have to _____ manually."
- "The moment this breaks, someone will feel pain because _____."
- Resist reading documentation until you can answer these. If you can't, the documentation will be noise.

**Actions:**
1. State the problem in one plain-language sentence before describing the solution.
2. Anchor every subsequent technical claim back to this sentence.

**Decision Point:** You can complete the sentence:
- "This component exists so that [person] does not have to [painful thing]."

---

### Step 2: Identify the Shape of the Data

**Goal:** Determine what kind of data this component produces, consumes, or transforms — because the shape of data defines the shape of all possible queries and correlations.

**Thinking Framework — The Four Data Shapes:**

| Shape | Description | Example Systems |
|---|---|---|
| **Number over time** | A value sampled at regular intervals | Prometheus, CloudWatch metrics |
| **Event stream** | Ordered text records, one per occurrence | Loki, CloudWatch Logs, Elasticsearch |
| **Request tree** | A hierarchy of spans, all sharing one ID | Tempo, Jaeger, Zipkin |
| **State snapshot** | Current desired vs. actual state of objects | Kubernetes API, CMDB |

**Key Questions to Ask:**
- Is this data a number, a string, a tree, or a graph?
- What is the cardinality — few values or millions of unique keys?
- What is the retention need — seconds, days, years?
- Is this append-only or mutable?

**Decision Point:** You can complete the sentence:
- "This component stores/produces [shape] data, which means it can answer [type of question] but cannot answer [type of question]."

**Why this matters:** The shape determines the blind spots. Prometheus can tell you the P99 latency over the last hour but cannot tell you why request #4821 specifically was slow. Tempo can tell you why request #4821 was slow but cannot tell you the overall P99. Knowing the shape tells you where to look and where not to.

---

### Step 3: Trace the Data Flow — Find the Breaks

**Goal:** Map the full lifecycle of data from birth to query, and identify every point where data disappears, is not captured, or cannot be correlated.

**Thinking Framework — Follow the Data:**
```
Something happens in the world
    → Who/what observes it?
    → How is it encoded?
    → How is it transmitted?
    → Who enriches or transforms it?
    → Where is it stored?
    → Who can query it?
    → What can they NOT see from here?
```

**Actions:**
1. Draw or describe the data flow as a pipeline, not a static diagram.
2. At each stage, explicitly ask: "What is lost here?"
3. Look for configuration that opts out of instrumentation (disabled flags, missing sidecars, absent ServiceMonitors) — these are the breaks.
4. Classify each break by severity:
   - **Critical:** Core functionality is a blind spot (e.g., the main orchestrator emits no telemetry)
   - **High:** Most services missing a full signal type
   - **Medium:** Signals exist but are disconnected (can't correlate A to B)
   - **Low:** Enrichment gaps (data exists but lacks context labels)

**Decision Point:** You have a list of breaks ranked by severity. Each break has:
- Where data disappears
- What configuration or code causes it
- What an engineer cannot know as a result

---

### Step 4: Separate Envelope from Contents

**Goal:** Distinguish between infrastructure-generated telemetry (what the platform knows about your service) and application-generated telemetry (what your service knows about itself).

**The Envelope vs. Contents Mental Model:**

```
ENVELOPE (platform-generated):
  The platform observes your service from the outside.
  It knows: request arrived, response sent, how long it took, status code.
  It does NOT know: what the request contained, why it was slow,
                    what business logic ran, what the LLM returned.

  Examples: Istio metrics, Kubernetes kube-state-metrics,
            load balancer access logs, VPC flow logs.

CONTENTS (application-generated):
  Your service reports on its own internal state.
  It knows: which database query ran, what the confidence score was,
            how many tokens the LLM consumed, which code path was taken.

  Examples: custom Prometheus counters, OTel trace spans,
            structured application logs, business event metrics.
```

**Key Questions to Ask:**
- For each service: does observability come from the envelope, the contents, or both?
- If only envelope: you know there is a problem, but not why.
- If only contents: you understand individual requests but may miss system-wide patterns.

**Thinking Framework:**
- "The envelope tells you there IS a problem."
- "The contents tell you WHY there is a problem."
- A mature observability stack needs both for every critical service.

**Actions:**
1. For each service in the system, mark: envelope-only / contents-only / both / neither.
2. Services that are "envelope-only" are where the next instrumentation investment should go.
3. Services that are "neither" are critical gaps — prioritize immediately.

**Decision Point:** You have a table of services with their coverage type. You can say:
- "We have envelope for all services but contents for only [N] of [M] services."

---

### Step 5: Apply the Three-Level Detective Test

**Goal:** Validate whether the observability stack (or any information architecture) can answer questions at all three levels of diagnosis. This is the completeness check.

**The Three Levels:**

```
Level 1 — "Is the system healthy?" (answered by Metrics / Numbers)
  Q: What is the current error rate?
  Q: Is P99 latency within SLA?
  Q: Are all pods running?
  Tool: Prometheus dashboards, alerts

Level 2 — "Where is it unhealthy?" (answered by Traces / Trees)
  Q: For this slow request, which service was the bottleneck?
  Q: Which Temporal activity failed and caused the retry?
  Q: What was the call graph for case ID 9876?
  Tool: Distributed tracing (Tempo, Jaeger)

Level 3 — "Why is it unhealthy?" (answered by Logs / Events)
  Q: What error message was printed during that span?
  Q: What was the exact SQL query that timed out?
  Q: What did the LLM API return before the timeout?
  Tool: Log aggregation (Loki, CloudWatch Logs)
```

**Scoring:**
- All three levels answerable → Observability is complete for this system
- Level 1 only → You know something is wrong, but you are guessing at cause
- Level 1 + Level 3 → You have raw evidence but no map to connect it
- Level 2 missing → You cannot trace individual requests; debugging is manual reconstruction

**The Cross-Signal Bonus (Level 4):**
When the three level