dotnet-trace-collect

What this skill does

# .NET Trace Collect

This skill helps developers diagnose production performance issues by recommending the right diagnostic tools for their environment, guiding data collection, and suggesting analysis approaches. It does not analyze code for anti-patterns or perform the analysis itself.

## When to Use

- A developer needs to investigate a production performance issue (high CPU, memory leak, slow requests, excessive GC, networking errors, etc.)
- Choosing the right diagnostic tool for a specific runtime, OS, or deployment topology
- Setting up and running diagnostic tool commands for data collection
- Understanding trade-offs between available tools (e.g. PerfView vs dotnet-trace)
- Collecting diagnostics from containerized or Kubernetes workloads

## When Not to Use

- Reviewing source code for performance anti-patterns (use a code review skill instead)
- Benchmarking during development (e.g. BenchmarkDotNet setup)
- Analyzing collected trace or dump files (this skill recommends tools for analysis, but does not perform it)

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| Symptom | Yes | What the developer is observing (high CPU, memory growth, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.) |
| Runtime | Yes | .NET Framework or modern .NET (and version, especially whether .NET 10+) |
| OS | Yes | Windows or Linux |
| Deployment | Yes | Non-container, container, or Kubernetes |
| Admin privileges | Recommended | Whether the developer has admin/root access on the target machine |
| Repro characteristics | Recommended | Whether the issue is easy to reproduce or requires a long time to manifest |

## Workflow

### Step 1: Understand the environment

Determine or ask the developer to clarify:

1. **Symptom**: What they are observing (high CPU, memory leak, slow requests, hangs, excessive GC, HTTP 5xx errors, networking timeouts, connection failures, assembly loading failures, etc.)
2. **Runtime**: .NET Framework or modern .NET? If modern .NET, which version? (Especially whether .NET 10 or later.)
3. **OS**: Windows or Linux?
4. **Deployment**: Running directly on the host, in a container, or in Kubernetes?
5. **Admin privileges**: Do they have admin/root access on the target machine or container?
6. **Repro characteristics**: Does the issue reproduce quickly, or does it take a long time to manifest?
7. **Workload context**: Determine or ask the user if you are running in the context of the workload (i.e., on the same machine or connected to the same environment where the issue is occurring). If so, you can run diagnostic commands directly on their behalf. If not, provide the commands as guidance for the user to run themselves.

Use this information to select the right tool in Step 2.

### Step 2: Recommend diagnostic tools

Select tools based on the environment using the priority rules below. Once a tool is selected, load the corresponding reference file for detailed command-line usage.

#### Tool reference lookup

| Environment | Reference file(s) |
|-------------|-------------------|
| Windows + modern .NET + admin | `references/perfview.md` |
| Windows + modern .NET, no admin | `references/dotnet-trace-collect.md` |
| Windows + .NET Framework | `references/perfview.md` |
| Linux + .NET 10+ + root | `references/dotnet-trace-collect-linux.md` |
| Linux + pre-.NET 10 | `references/dotnet-trace-collect.md` |
| Linux + native stacks needed | `references/perfcollect.md` |
| Container/K8s (console access) | `references/dotnet-trace-collect.md` (or `dotnet-trace-collect-linux.md`) |
| Container/K8s (no console) | `references/dotnet-monitor.md` |

#### Quick decision matrix (first-pass triage)

| Environment | Preferred tool | Fallback / Notes |
|-------------|----------------|------------------|
| Windows + modern .NET + admin | PerfView | If admin is unavailable, use `dotnet-trace` |
| Windows + .NET Framework + admin | PerfView | Without admin, there is no trace fallback; for hangs/memory leaks, provide dump commands directly (`procdump -ma` or Task Manager) since `dump-collect` does not support .NET Framework |
| Linux + .NET 10+ + root | `dotnet-trace collect-linux` | Use `dotnet-trace` if root or kernel prerequisites are not met |
| Linux + pre-.NET 10 | `dotnet-trace` | Add `perfcollect` when native stacks are needed (requires root) |
| Linux container/Kubernetes | Console tools if in workload context; `dotnet-monitor` if no console access | See Linux Container / Kubernetes section for details |

#### Windows (non-container, modern .NET)

1. **PerfView** (preferred) — produces richer ETW-based data; requires admin privileges. For **slow requests**, add `/ThreadTime` to capture thread-level wait and block detail.
2. **`dotnet-trace`** — fallback when admin privileges are not available.
3. For **long-running repros**: use PerfView with a `/StopOn` trigger that fires on the **symptom you want to capture** (e.g., `/StopOnPerfCounter`, `/StopOnGCEvent`, `/StopOnException`) and a circular buffer (`/CircularMB` + `/BufferSizeMB`). **Critical: the stop trigger must fire on the interesting event, not the recovery.** The circular buffer continuously overwrites old data, so if you trigger on recovery, the buffer may have already overwritten the interesting behavior by the time collection stops. Only add `/StartOn` if the start event is known to precede the stop event. For **slow requests**, do not include a stop trigger by default — let the user design one based on their specific scenario.

#### Windows containers

1. **PerfView** — most Windows containers (including Kubernetes on Windows) use process-isolation by default. Collect from the host with `/EnableEventsInContainers`. After collection, you have two options:
   - **Analyze locally while the container is still running** — PerfView can reach into the live container to resolve symbols, so you can open the trace immediately on the host machine.
   - **Analyze off-machine** — before the container shuts down, copy the `.etl.zip` into the container and run `PerfViewCollect merge /ImageIDsOnly` inside it to embed symbol information. Then copy the merged trace out. Without this merge step, symbols for binaries inside the container will be unresolvable on other machines.

   For the less common Hyper-V containers, collect inside the container directly. See [references/perfview.md](references/perfview.md) for detailed commands.
2. **`dotnet-monitor`**, **`dotnet-trace`** — inside the container if the tools are installed in the image. For dumps, invoke the **`dump-collect`** skill.

#### Windows (.NET Framework)

1. **PerfView** — the primary diagnostic tool for .NET Framework on Windows. Requires admin.
2. Same trigger guidance for long repros: use `/StopOn` triggers that fire on the symptom (e.g., `/StopOnPerfCounter`, `/StopOnGCEvent`, `/StopOnException`) with `/CircularMB` + `/BufferSizeMB`.
3. **Without admin**: PerfView requires admin, and there are no alternative trace tools for .NET Framework. Process dumps can still be captured without admin — provide dump commands directly (e.g., `procdump -ma <PID>` or Task Manager) since the `dump-collect` skill does not support .NET Framework. Dumps can help diagnose hangs and memory leaks. However, for **high CPU**, **slow requests**, and **excessive GC**, there is no way to investigate on .NET Framework without admin access. Advise the user to obtain admin privileges.

#### Linux (non-container, .NET 10+)

1. **`dotnet-trace collect-linux`** (preferred) — uses `perf_events` for richer traces including native call stacks and kernel events. Captures machine-wide by default (no PID required). Requires root and kernel >= 6.4.
2. **`dotnet-trace`** — fallback when root privileges are not available or kernel requirements are not met. Managed stacks only.

#### Linux (non-container, pre-.NET 10)

1. **`dotnet-trace`** (preferred) — managed trace collection;