ln-811-performance-profiler

What this skill does

> **Paths:** File paths (`references/`, `../ln-*`) are relative to this skill directory.

# ln-811-performance-profiler

**Type:** L3 Worker
**Category:** 8XX Optimization

Runtime profiler that executes the optimization target, measures multiple metrics (CPU, memory, I/O, time), instruments code for per-function breakdown, and produces a standardized performance map from real data.

---

## Overview

| Aspect | Details |
|--------|---------|
| **Input** | Problem statement: target (file/endpoint/pipeline) + observed metric |
| **Output** | Performance map (multi-metric, per-function), suspicion stack, bottleneck classification |
| **Pattern** | Discover test → Baseline run → Static analysis → Deep profile → Performance map → Report |

---

## Workflow

**Phases:** Test Discovery → Baseline Run → Static Analysis → Deep Profile → Performance Map → Report

---

## Phase 0: Test Discovery/Creation

**MANDATORY READ:** Load `references/ci_tool_detection.md` for test framework detection.
**MANDATORY READ:** Load `references/benchmark_generation.md` for auto-generating benchmarks when none exist.

Find or create commands that exercise the optimization target. Two outputs: `test_command` (profiling/measurement) and `e2e_test_command` (functional safety gate).

### Step 1: Discover test_command

| Priority | Method | Action |
|----------|--------|--------|
| 1 | User-provided | User specifies test command or API endpoint |
| 2 | Discover existing E2E test | Grep test files for target entry point (stop at first match) |
| 3 | Create test script | Generate per `references/benchmark_generation.md` to `.hex-skills/optimization/{slug}/profile_test.sh` |

**E2E discovery protocol** (stop at first match):

| Priority | Method | How |
|----------|--------|-----|
| 1 | Route-based search | Grep e2e/integration test files for entry point route |
| 2 | Function-based search | Grep for entry point function name |
| 3 | Module-based search | Grep for import of entry point module |

**Test creation** (if no existing test found):

| Target Type | Generated Script |
|-------------|-----------------|
| API endpoint | `curl -w "%{time_total}" -o /dev/null -s {endpoint}` |
| Function | Stack-specific benchmark per `references/benchmark_generation.md` |
| Pipeline | Full pipeline invocation with test input |

### Step 2: Discover e2e_test_command

If `test_command` came from E2E discovery (Step 1 priority 2): `e2e_test_command = test_command`.

Otherwise, run E2E discovery protocol again (same 3-priority table) to find a separate functional safety test.

If not found: `e2e_test_command = null`, log: `WARNING: No e2e test covers {entry_point}. Full test suite serves as functional gate.`

### Output

| Field | Description |
|-------|-------------|
| `test_command` | Command for profiling/measurement |
| `e2e_test_command` | Command for functional safety gate (may equal test_command, or null) |
| `e2e_test_source` | Discovery method: user / route / function / module / none |

---

## Phase 1: Baseline Run (Multi-Metric)

Run `test_command` with system-level profiling. Capture simultaneously:

| Metric | How to Capture | When |
|--------|---------------|------|
| Wall time | `time` wrapper or test harness | Always |
| CPU time (user+sys) | `/usr/bin/time -v` or language profiler | Always |
| Memory peak (RSS) | `/usr/bin/time -v` (Max RSS) or `tracemalloc` / `process.memoryUsage()` | Always |
| I/O bytes | `/usr/bin/time -v` or structured logs | If I/O suspected |
| HTTP round-trips | Count from structured logs or application metrics | If network I/O in call graph |
| GPU utilization | `nvidia-smi --query-gpu` | Only if CUDA/GPU detected in stack |

### Baseline Protocol

| Parameter | Value |
|-----------|-------|
| Runs | 3 |
| Metric | Median |
| Warm-up | 1 discarded run |
| Output | `baseline` — multi-metric snapshot |

### Monitor for Profiler Runs (Claude Code 2.1.98+)

**MANDATORY READ:** Load `references/monitor_integration_pattern.md`

During baseline and deep profile runs:
`Monitor(command="{test_command} 2>&1", timeout_ms=300000, description="profiler run N")`

Detect crashes or infinite loops immediately. Fallback: `Bash` with `timeout`.

---

## Phase 2: Static Analysis → Instrumentation Points

**MANDATORY READ:** Load [bottleneck_classification.md](references/bottleneck_classification.md)

Trace call chain from code + build suspicion stack. **Purpose:** guide WHERE to instrument in Phase 3.

### Step 1: Trace Call Chain

Starting from entry point, trace depth-first (max depth 5). At each step, READ the full function body.

**Cross-service tracing:** If `service_topology` is available from coordinator and a step makes an HTTP/gRPC call to another service whose code is accessible:

| Situation | Action |
|-----------|--------|
| HTTP call to service with code in submodule/monorepo | Follow into that service's handler: resolve route → trace handler code (depth resets to 0 for the new service) |
| HTTP call to service without accessible code | Classify as External, record latency estimate |
| gRPC/message queue to known service | Same as HTTP — follow into handler if code accessible |

Record `service: "{service_name}"` on each step to track which service owns it. The performance_map `steps` tree can span multiple services.

**Depth-First Rule:** If code of the called service is accessible — ALWAYS profile INSIDE. NEVER classify an accessible service as "External/slow" without profiling its internals. "Slow" is a symptom, not a diagnosis.

**5 Whys for each bottleneck:** Before reporting a bottleneck, chain "why?" until you reach config/architecture level:
1. "What is slow?" → alignment service (5.9s) 2. "Why?" → 6 pairs × ~1s each 3. "Why ~1s per pair?" → O(n²) mwmf computation 4. "Why O(n²)?" → library default, not production config 5. "Why default?" → `matching_methods` not configured → **root cause = config**

### Step 2: Classify & Suspicion Scan

For each step, classify by type (CPU, I/O-DB, I/O-Network, I/O-File, Architecture, External, Cache) and scan for performance concerns.

Suspicion checklist (**minimum, not limitation**):

| Category | What to Look For |
|----------|-----------------|
| Connection management | Client created per-request? Missing pooling? Missing reuse? |
| Data flow | Data read multiple times? Over-fetching? Unnecessary transforms? |
| Async patterns | Sync I/O in async context? Sequential awaits without data dependency? |
| Resource lifecycle | Unclosed connections? Temp files? Memory accumulation in loop? |
| Configuration | Hardcoded timeouts? Default pool sizes? Missing batch size config? |
| Redundant work | Same validation at multiple layers? Same data loaded twice? |
| Architecture | N+1 in loop? Batch API unused? Cache infra unused? Sequential-when-parallel? |
| *(open)* | Anything else spotted — checklist does not limit findings |

### Step 2b: Suspicion Deduplication

**MANDATORY READ:** Load `references/output_normalization.md`

After generating suspicions across all call chain steps, normalize and deduplicate per §1-§2:
- Normalize suspicion descriptions (replace specific values with placeholders)
- Group identical suspicions across different steps → merge into single entry with `affected_steps: [list]`
- Example: "Missing connection pooling" found in steps 1.1, 1.2, 1.3 → one suspicion with `affected_steps: ["1.1", "1.2", "1.3"]`

### Step 3: Verify & Map to Instrumentation Points

```
FOR each suspicion:
  1. VERIFY: follow code to confirm or dismiss
  2. VERDICT: CONFIRMED → map to instrumentation point | DISMISSED → log reason
  3. For each CONFIRMED suspicion, identify:
     - function to wrap with timing
     - I/O call to count
     - memory allocation to track
```

### Profiler Selection (per stack)

| Stack | Non-invasive profiler | Invasive (if non-invasive insufficient) |
|-------|----------------------|----------------------------------------|
| Python | `py-spy`, `cProfile` | `time.perf_counter()` decorators |
|