build-jvm-analysis

What this skill does

# JVM Analysis

Patterns for analyzing, optimizing, and debugging JVM applications — both static and runtime.

## Philosophy

1. **Measure before tuning** — profile first, optimize based on evidence
2. **Static + runtime complement** — static finds structural issues, runtime finds behavioral issues
3. **Production-safe tooling** — low overhead profilers that won't crash or slow production
4. **Understand tool limitations** — safepoint bias, entry point coverage, what tools can't see
5. **Right tool for the job** — no single tool does everything

---

## Static Analysis

### Dead Code Detection

```
Finding unused code?
├── Have production traffic → Runtime analysis (Scavenger)
├── Need static-only analysis →
│   ├── Simple → ProGuard -printusage
│   └── Custom analysis → SootUp or ProGuard Core
└── Just bug patterns → SpotBugs, Error Prone
```

| Tool | Type | What It Does | When to Use |
|------|------|--------------|-------------|
| Scavenger | Runtime | Tracks actual usage in production | Have production data |
| ProGuard -printusage | Static | Lists unreachable from entry points | Know your entry points |
| SootUp | Library | Call graphs, data flow analysis | Building custom analysis |
| ProGuard Core | Library | Bytecode analysis primitives | Building custom tools |
| Dead Code Agent | Runtime | Tracks class loading | Quick prototype |

**Key insight:** No turnkey "run this, get dead code" CLI exists. You either:
- Deploy runtime agents (needs production) — Scavenger
- Configure static analysis (needs entry points) — ProGuard
- Build custom tooling (needs engineering) — SootUp

### SootUp for Code Analysis

SootUp provides call graph construction and analysis primitives. Useful for:
- Building reachability analysis (find what's called from entry points)
- Data flow analysis (track how values propagate)
- Dependency mapping (what calls what)

*(Karakaya et al., TACAS 2024)*

**Gotcha:** SootUp is a library, not a tool. You build analysis on top of it.

### Bug Detection (Not Dead Code)

| Tool | Focus | Integration |
|------|-------|-------------|
| SpotBugs | Bug patterns (400+) | Gradle/Maven, CI |
| Error Prone | Compile-time checks | javac plugin |
| NullAway | Null safety | Error Prone plugin |

---

## Runtime Analysis

### Profiler Selection

```
Need production profiling?
├── YES → CPU or memory?
│   ├── CPU → Need flame graphs?
│   │   ├── YES → async-profiler
│   │   └── NO → JFR (built-in, zero config)
│   └── Memory → JFR (allocation profiling)
└── NO (development only) → VisualVM or IntelliJ Profiler
```

| Profiler | Overhead | Safepoint-Free | Output | Tradeoff | Best For |
|----------|----------|----------------|--------|----------|----------|
| async-profiler | ~2% CPU | Yes | Flame graphs, JFR | Requires native agent attachment | Production CPU/allocation |
| JFR + JMC | ~1-2% | Partial (improved Java 16+) | Binary events | Less granular CPU data | Continuous monitoring |
| VisualVM | 5-10% | No | Various | Safepoint bias distorts results | Development only |
| IntelliJ Profiler | ~2% | Yes (uses async-profiler) | Flame graphs | IDE dependency | IDE-integrated |

*(InfoQ 2025)*

**Why safepoint-free matters:** JVM can only safely inspect threads at safepoints. JVMTI-based profilers (VisualVM, hprof) miss code between safepoints, skewing flame graphs toward safepoint-heavy code. async-profiler uses `AsyncGetCallTrace` to sample anytime (Wakart 2016).

### Garbage Collector Selection

```
Heap size?
├── < 4 GB → G1 (default since JDK 9)
│   WHY: ZGC/Shenandoah overhead not justified; G1's region-based collection efficient at this scale
├── 4-32 GB → Latency-sensitive?
│   ├── YES → ZGC or Shenandoah
│   │   WHY: Concurrent marking/compaction keeps pauses <10ms regardless of heap size
│   └── NO → G1
│       WHY: G1's mixed collections handle this range well; simpler tuning
└── > 32 GB → ZGC (generational, JDK 21+)
    WHY: ZGC's concurrent compaction scales linearly; G1 pauses grow with heap
```

| Collector | Pause Target | Heap Size | Tradeoff | JDK | Best For |
|-----------|--------------|-----------|----------|-----|----------|
| G1GC | 200ms (tunable) | Any | Pauses scale with heap | 9+ default | General workloads |
| ZGC | <1ms | Large (100GB+) | ~15% throughput cost vs G1 | 15+ prod, 21+ gen | Latency-critical |
| Shenandoah | <10ms | Large | Higher CPU for barriers | 12+ (Red Hat) | Low-latency, older JDKs |
| Parallel | Max throughput | Medium | Stop-the-world only | All | Batch processing |

**Why the thresholds:**
- **<4GB**: G1's region-based approach (2048 regions default) works well. ZGC's colored pointers and load barriers add overhead not justified at small scale (Oracle GC Tuning Guide).
- **4-32GB**: The "compressed OOPs" boundary. Above 32GB, object pointers expand from 4 to 8 bytes, increasing memory footprint ~20%. ZGC handles this better (Shipilev 2019).
- **>32GB**: G1's pause times grow with live set size during mixed collections. ZGC's concurrent compaction maintains <1ms regardless (ZGC wiki, Oracle).

**Key insight:** ZGC generational (JDK 21+) closes the throughput gap — concurrent minor collections reduce allocation pressure (JEP 439).

*(Oracle GC Tuning Guide, Shipilev JVM Anatomy Quarks, JEP 439)*

### Heap Dump Analysis

```
OOM or suspected leak?
├── Capture dump → -XX:+HeapDumpOnOutOfMemoryError
├── Analyze → Eclipse MAT or HeapHero
│   ├── Run "Leak Suspects" report
│   ├── Check retained heap (not just shallow)
│   └── Path to GC Roots (exclude weak refs)
└── Fix → Collections holding references, static fields, caches without eviction
```

### Thread Dump Analysis

```
Application hanging or slow?
├── Capture → jstack -l <pid> (or jcmd <pid> Thread.print)
├── Take 3-5 dumps seconds apart
├── Analyze:
│   ├── BLOCKED threads → lock contention
│   ├── WAITING on same monitor → bottleneck
│   └── Same stack across dumps → stuck thread
└── Tools: FastThread.io, TDA, or manual grep
```

## Production Gotchas

### Safepoint Bias

- **Trap**: Traditional profilers (VisualVM, hprof) only sample at safepoints
- **Impact**: Misleading flame graphs — hot spots skewed to safepoint-heavy code
- **Detection**: Compare async-profiler output vs traditional profiler
- **Fix**: Use async-profiler or JFR (Java 16+ with JEP 376)

**Why it happens:** JVM can only safely inspect thread state at safepoints — points where all threads are known to be in a consistent state. JVMTI's `GetStackTrace` requires this. Safepoints occur at method returns, loop back-edges, and allocation. Tight loops without allocations may run millions of cycles between safepoints, becoming invisible (Wakart 2015).

*(Wakart 2015, async-profiler docs)*

### DebugNonSafepoints Flag

- **Trap**: Even async-profiler needs `-XX:+DebugNonSafepoints` for accurate frame resolution
- **Impact**: Inlined methods may not appear in profiles
- **Fix**: Start JVM with flag, or attach agent early

```bash
# At JVM start (recommended)
java -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -agentpath:/path/to/libasyncProfiler.so ...

# Late attach works but misses already-compiled methods
```

### Container Memory Limits (OOMKilled)

- **Trap**: JVM uses more memory than `-Xmx` (native memory, metaspace, stacks, codecache)
- **Impact**: Kubernetes kills pod with OOMKilled even when heap looks fine
- **Detection**: `kubectl describe pod` shows OOMKilled; native memory tracking shows usage
- **Fix**: Budget 25-30% of container memory for non-heap

```bash
# Good: percentage-based, container-aware
java -XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport ...

# Budget breakdown for 2GB container:
# Heap: ~1.5GB (75%)
# Metaspace: ~100MB (default MaxMetaspaceSize unbounded, set explicitly)
# Thread stacks: ~100MB (100 threads × 1MB default Linux stack)
# CodeCache: ~50MB (240MB reserved, typically uses ~50MB)
# Native/JNI: ~150MB buffer (JDBC drivers, compression libs, etc.)
```

**Why 25-30%:** Empirical guidance from production incidents. Exact overhead de