configuring-opentelemetry-dotnet

What this skill does

# Configuring OpenTelemetry in .NET

## When to Use

- Adding distributed tracing to an ASP.NET Core application
- Setting up OpenTelemetry exporters (OTLP is the primary protocol; Jaeger accepts OTLP natively; Prometheus OTLP ingestion requires explicit opt-in)
- Creating custom metrics or trace spans for business operations
- Troubleshooting distributed trace context propagation across services

## When Not to Use

- The user wants application-level logging only (use ILogger, Serilog)
- The user is using Application Insights SDK directly (different API)
- The user needs APM with a commercial vendor's proprietary SDK

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| ASP.NET Core project | Yes | The application to instrument |
| Observability backend | No | Where to export: OTLP collector, Aspire dashboard, Jaeger (accepts OTLP natively) |

## Workflow

### Step 1: Install the correct packages

**There are many OpenTelemetry NuGet packages. Install exactly these:**

```bash
# Core SDK + ASP.NET Core instrumentation + logging integration
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

# Exporter
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol  # OTLP exporter for traces, metrics, AND logs

# Optional — dev/local debugging only (do NOT include in production deployments)
# dotnet add package OpenTelemetry.Exporter.Console
```

**Do NOT install `OpenTelemetry` alone** — you need `OpenTelemetry.Extensions.Hosting` for proper DI integration.

#### Optional: additional auto-instrumentation packages

Install only the packages that match the libraries your application uses:

```bash
dotnet add package OpenTelemetry.Instrumentation.SqlClient           # SQL Server queries
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore  # EF Core
dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient       # gRPC calls
dotnet add package OpenTelemetry.Instrumentation.Runtime             # GC, thread pool metrics
```

### Step 2: Configure all signals in Program.cs

```csharp
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: builder.Environment.ApplicationName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            // Filter out health check endpoints from traces
            options.Filter = httpContext =>
                !httpContext.Request.Path.StartsWithSegments("/healthz");
        })
        .AddHttpClientInstrumentation(options =>
        {
            options.RecordException = true;
        })
        // Optional: add SQL instrumentation if using SqlClient directly
        // .AddSqlClientInstrumentation(options =>
        // {
        //     options.SetDbStatementForText = true;
        //     options.RecordException = true;
        // })
        // Custom activity sources (must match ActivitySource names in your code)
        .AddSource("MyApp.Orders")
        .AddSource("MyApp.Payments")
        .AddSource("MyApp.Messaging"))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        // Optional: .AddRuntimeInstrumentation() for GC and thread pool metrics
        //   (requires OpenTelemetry.Instrumentation.Runtime package)
        // Custom meters (must match Meter names in your code)
        .AddMeter("MyApp.Metrics"))
    .WithLogging(logging =>
    {
        logging.IncludeScopes = true;
        // logging.IncludeFormattedMessage = true;  // Enable if you need the formatted message string in log exports
    })
    // Single OTLP exporter for all signals — reads OTEL_EXPORTER_OTLP_ENDPOINT
    // env var (defaults to http://localhost:4317). Override via environment variable
    // or appsettings.json configuration.
    .UseOtlpExporter();
```

### Step 3: Understanding log–trace correlation

The `.WithLogging()` call in Step 2 integrates ILogger with OpenTelemetry:

- Each log entry automatically includes TraceId and SpanId for correlation with traces
- The service resource from `.ConfigureResource()` propagates to logs automatically
- `UseOtlpExporter()` applies to logs alongside traces and metrics
- No additional packages or separate `SetResourceBuilder` call needed

### Step 4: Create custom spans (Activities) for business operations

```csharp
using System.Diagnostics;
using Microsoft.Extensions.Logging;

public class OrderService
{
    // Create an ActivitySource matching what you registered in Step 2
    private static readonly ActivitySource ActivitySource = new("MyApp.Orders");
    private readonly ILogger<OrderService> _logger;

    public OrderService(ILogger<OrderService> logger) => _logger = logger;

    public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
    {
        // Start a new span
        using var activity = ActivitySource.StartActivity("ProcessOrder");

        // Add attributes (tags) to the span
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);

        try
        {
            // Child span for validation
            using (var validationActivity = ActivitySource.StartActivity("ValidateOrder"))
            {
                await ValidateOrderAsync(request);
                validationActivity?.SetTag("validation.result", "passed");
            }

            // Child span for payment
            using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment",
                ActivityKind.Client))  // Client = outgoing call
            {
                paymentActivity?.SetTag("payment.method", request.PaymentMethod);
                await ProcessPaymentAsync(request);
            }

            var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId, Status = "Completed" };

            activity?.SetTag("order.status", "completed");
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            // Log via ILogger — OpenTelemetry captures this with trace correlation.
            // Prefer logging over activity.RecordException() as OTel is deprecating
            // span events for exception recording in favor of log-based exceptions.
            _logger.LogError(ex, "Order processing failed for customer {CustomerId}", request.CustomerId);
            throw;
        }
    }
}
```

**Critical: `ActivitySource` name must match `AddSource("...")` in configuration.** Unmatched sources are silently ignored — this is the #1 debugging issue.

### Step 5: Create custom metrics

Use `IMeterFactory` (injected via DI) to create meters — this ensures proper lifetime management and testability.

```csharp
using System.Diagnostics;
using System.Diagnostics.Metrics;

public class OrderMetrics
{
    private readonly Counter<long> _ordersProcessed;
    private readonly Histogram<double> _orderProcessingDuration;
    private readonly UpDownCounter<int> _activeOrders;

    public OrderMetrics(IMeterFactory meterFactory)
    {
        // Meter name must match AddMeter("...") in configuration
        var meter = meterFactory.Create("MyApp.Metrics");

        // Counter — use for things that only go up
        _ordersProcessed = meter.CreateCounter<long>(
            "orders.processed", "orders", "Total orders successfully processed");

        // Histogram — use for measuring distributions (latency, sizes)
        _orderProcessingDuration = meter.CreateHistogram<double>(
            "orders.processing_duration", "ms", "Time to process an order