dotnet-observability

What this skill does

# dotnet-observability

Modern observability for .NET applications using OpenTelemetry, structured logging, health checks, and custom metrics. Covers the three pillars of observability (traces, metrics, logs), integration with `Microsoft.Extensions.Diagnostics` and `System.Diagnostics`, and production-ready health check patterns.

**Out of scope:** DI container mechanics and service lifetimes -- see [skill:dotnet-csharp-dependency-injection]. Async/await patterns -- see [skill:dotnet-csharp-async-patterns]. Testing observability output -- see [skill:dotnet-integration-testing] for verifying telemetry in integration tests. CI/CD pipeline integration for telemetry collection -- see [skill:dotnet-gha-patterns] and [skill:dotnet-ado-patterns]. Middleware pipeline patterns (request logging middleware, exception handling middleware) -- see [skill:dotnet-middleware-patterns].

Cross-references: [skill:dotnet-csharp-dependency-injection] for service registration, [skill:dotnet-csharp-async-patterns] for async patterns in background exporters, [skill:dotnet-resilience] for Polly telemetry integration, [skill:dotnet-middleware-patterns] for request/exception logging middleware.

---

## OpenTelemetry Setup

OpenTelemetry is the standard observability framework in .NET. The .NET SDK includes native support for `System.Diagnostics.Activity` (traces) and `System.Diagnostics.Metrics` (metrics), which OpenTelemetry collects and exports.

### Package Landscape

| Package | Purpose |
|---------|---------|
| `OpenTelemetry.Extensions.Hosting` | Host integration, lifecycle management |
| `OpenTelemetry.Instrumentation.AspNetCore` | Automatic HTTP server trace/metric instrumentation |
| `OpenTelemetry.Instrumentation.Http` | Automatic `HttpClient` trace/metric instrumentation |
| `OpenTelemetry.Instrumentation.Runtime` | GC, thread pool, assembly metrics |
| `OpenTelemetry.Exporter.OpenTelemetryProtocol` | OTLP exporter (gRPC/HTTP) for collectors |
| `OpenTelemetry.Exporter.Console` | Console exporter for local development |

Install the core stack:

```xml
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.*" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.*" />
```

### Aspire Service Defaults Integration

If using .NET Aspire, the `ServiceDefaults` project configures OpenTelemetry automatically. This is the recommended approach for Aspire apps -- do not duplicate this configuration manually:

```csharp
// ServiceDefaults/Extensions.cs (generated by Aspire)
public static IHostApplicationBuilder AddServiceDefaults(
    this IHostApplicationBuilder builder)
{
    builder.ConfigureOpenTelemetry();
    builder.AddDefaultHealthChecks();
    // ... other defaults
    return builder;
}
```

For non-Aspire apps, configure OpenTelemetry explicitly as shown below.

### Full Configuration (Non-Aspire)

```csharp
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: builder.Environment.ApplicationName,
            serviceVersion: typeof(Program).Assembly
                .GetCustomAttribute<AssemblyInformationalVersionAttribute>()
                ?.InformationalVersion ?? "unknown"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("MyApp.*")          // Custom ActivitySources
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("MyApp.*")           // Custom Meters
        .AddOtlpExporter());
```

### OTLP Configuration via Environment Variables

The OTLP exporter reads standard environment variables -- no code changes needed between environments:

```bash
# Collector endpoint (gRPC default)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Or HTTP/protobuf
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Resource attributes
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=myapp

# Service name (overrides code-based configuration)
OTEL_SERVICE_NAME=order-api
```

---

## Distributed Tracing

### How .NET Tracing Works

.NET uses `System.Diagnostics.Activity` as its native tracing primitive. OpenTelemetry maps these to spans:

| .NET Concept | OpenTelemetry Concept |
|---|---|
| `ActivitySource` | Tracer |
| `Activity` | Span |
| `Activity.SetTag` | Span attribute |
| `Activity.AddEvent` | Span event |
| `Activity.SetStatus` | Span status |

### Custom Traces

```csharp
public sealed class OrderService
{
    // One ActivitySource per logical component, named after the namespace
    private static readonly ActivitySource s_activitySource = new("MyApp.Orders");

    public async Task<Order> CreateOrderAsync(
        CreateOrderRequest request,
        CancellationToken ct)
    {
        using var activity = s_activitySource.StartActivity(
            "CreateOrder",
            ActivityKind.Internal);

        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.line_count", request.Lines.Count);

        var order = new Order { /* ... */ };

        activity?.AddEvent(new ActivityEvent("OrderValidated"));

        await _db.Orders.AddAsync(order, ct);
        await _db.SaveChangesAsync(ct);

        activity?.SetTag("order.id", order.Id);
        activity?.SetStatus(ActivityStatusCode.Ok);

        return order;
    }
}
```

### Trace Context Propagation

W3C Trace Context is the default propagation format in .NET. It works automatically across HTTP boundaries with `HttpClient`:

```csharp
// Trace context is automatically propagated via traceparent/tracestate headers
// when using HttpClient with OpenTelemetry.Instrumentation.Http.
// No manual propagation needed for HTTP-based communication.
```

For message-based communication (queues, event buses), propagate context explicitly:

```csharp
// Producer: inject context into message headers
var propagator = Propagators.DefaultTextMapPropagator;
var carrier = new Dictionary<string, string>();
var currentActivity = Activity.Current;
if (currentActivity is not null)
{
    propagator.Inject(
        new PropagationContext(currentActivity.Context, Baggage.Current),
        carrier,
        (dict, key, value) => dict[key] = value);
}
// Attach carrier as message headers

// Consumer: extract context from message headers
var parentContext = propagator.Extract(
    default,
    messageHeaders,
    (headers, key) => headers.TryGetValue(key, out var value)
        ? [value] : []);

using var activity = s_activitySource.StartActivity(
    "ProcessMessage",
    ActivityKind.Consumer,
    parentContext.ActivityContext);
```

---

## Metrics

### Built-in Metrics

ASP.NET Core and HttpClient emit metrics automatically when OpenTelemetry instrumentation is configured:

| Meter | Key Metrics |
|-------|-------------|
| `Microsoft.AspNetCore.Hosting` | `http.server.request.duration`, `http.server.active_requests` |
| `Microsoft.AspNetCore.Routing` | `aspnetcore.routing.match_attempts` |
| `System.Net.Http` | `http.client.request.duration`, `http.client.active_requests` |
| `System.Runtime` | `process.runtime.dotnet.gc.collections.count`, `process.runtime.dotnet.threadpool.threads.count` |

### Custom Metrics

Use `System.Diagnostics.Metrics` for application-specific metrics:

```csharp
public sealed class OrderMetrics
{
    // One Meter per logical component
    private readonly Counter<long> _ordersCreated;
    private readonly Histogram<double> _orderProcessingDuration;
    priva