dotnet-structured-logging

What this skill does

# dotnet-structured-logging

Log pipeline design and operations for .NET distributed systems. Covers log aggregation architecture (ELK, Seq, Grafana Loki), structured query patterns for each platform, log sampling and volume management strategies, PII scrubbing and destructuring policies, and cross-service correlation beyond single-service log scopes. This skill addresses what happens _after_ log emission -- the pipeline, query, and operations layer.

**Out of scope:** Log emission mechanics (Serilog/NLog/MEL configuration, source-generated LoggerMessage, enrichers, single-service log scopes, sink registration, OTel logging export) -- see [skill:dotnet-observability]. Application configuration and options pattern -- see [skill:dotnet-csharp-configuration]. Distributed tracing setup and trace context propagation -- see [skill:dotnet-observability].

Cross-references: [skill:dotnet-observability] for log emission, Serilog/MEL configuration, and OpenTelemetry logging export, [skill:dotnet-csharp-configuration] for appsettings.json configuration patterns used in log pipeline setup.

---

## Log Aggregation Architecture

### Architecture Options

| Platform | Ingest | Storage | Query | Best for |
|----------|--------|---------|-------|----------|
| **ELK** (Elasticsearch, Logstash, Kibana) | Logstash / Filebeat | Elasticsearch | KQL in Kibana | Large-scale, flexible schema, full-text search |
| **Seq** | HTTP API / Serilog sink | Built-in | Seq signal expressions | .NET-native, developer-friendly, structured queries |
| **Grafana Loki** | Promtail / OTel Collector | Loki (label-indexed) | LogQL | Cost-effective, Grafana ecosystem, label-based queries |
| **Azure Monitor** | OTel Collector / Application Insights SDK | Log Analytics workspace | KQL (Kusto) | Azure-native, integrated alerting, cost management |

### Recommended Pipeline Patterns

**Pattern 1: OTel Collector as central router**

```
App (OTLP) --> OTel Collector --> Elasticsearch / Loki / Azure Monitor
                  |
                  +--> Sampling / filtering / PII scrub
```

The OpenTelemetry Collector acts as a vendor-neutral log router. Applications emit logs via OTLP; the collector handles filtering, sampling, enrichment, and routing to one or more backends. This decouples applications from backend choice.

```yaml
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  filter:
    logs:
      exclude:
        match_type: strict
        bodies:
          - "Health check endpoint hit"

exporters:
  elasticsearch:
    endpoints: ["https://es-cluster:9200"]
    logs_index: "app-logs"
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch, filter]
      exporters: [elasticsearch, loki]
```

**Pattern 2: Direct sink (smaller deployments)**

```
App (Serilog) --> Seq / Elasticsearch sink
```

For smaller systems or development environments, Serilog sinks write directly to the aggregation platform. This avoids the OTel Collector but couples the application to the backend.

### .NET Application OTLP Configuration

For .NET application-side OTLP log export configuration (`builder.Logging.AddOpenTelemetry()`), see [skill:dotnet-observability]. The OTLP endpoint is configured via environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`), keeping application code backend-agnostic.

---

## Structured Query Patterns

Structured logs store each property as a queryable field. The query syntax differs by platform but the concepts are consistent: filter by property name, value, severity, and time range.

### Kibana KQL (Elasticsearch / ELK)

```
# Find errors for a specific order
level: "Error" AND OrderId: "abc-123"

# Find slow operations (custom Duration property)
Duration > 5000 AND ServiceName: "order-api"

# Wildcard on message template
message: "Failed to process*" AND NOT level: "Debug"

# Time-scoped with correlation
TraceId: "0af7651916cd43dd8448eb211c80319c" AND @timestamp >= "2025-01-15T10:00:00"
```

### Seq Signal Expressions

```
# Find errors for a specific order
@Level = 'Error' and OrderId = 'abc-123'

# Find slow operations
Duration > 5000 and Application = 'order-api'

# Free-text search combined with structured filter
@Message like '%timeout%' and @Level in ['Warning', 'Error']

# Correlation across services
TraceId = '0af7651916cd43dd8448eb211c80319c'
```

Seq signals are saved queries that trigger alerts. Define signals for recurring patterns (e.g., "Payment failures > 10/min") and attach notification channels.

### Grafana LogQL (Loki)

```
# Filter by labels then regex on log line
{service_name="order-api"} |= "Error" | json | OrderId="abc-123"

# Structured field extraction and filtering
{service_name="order-api"} | json | Duration > 5000

# Count errors per service over time (for dashboards)
sum(rate({service_name=~".+"} |= "Error" [5m])) by (service_name)
```

### Azure Monitor KQL (Kusto)

```kusto
// Find errors for a specific order
traces
| where severityLevel >= 3
| where customDimensions.OrderId == "abc-123"
| order by timestamp desc

// Slow operations
traces
| where toint(customDimensions.Duration) > 5000
| where cloud_RoleName == "order-api"

// Cross-service correlation
union traces, exceptions
| where operation_Id == "0af7651916cd43dd8448eb211c80319c"
| order by timestamp asc
```

---

## Log Sampling and Volume Management

High-throughput systems can generate millions of log events per minute. Without sampling, storage costs and query performance degrade rapidly.

### Sampling Strategies

| Strategy | How it works | Use when |
|----------|-------------|----------|
| **Head-based** | Decide to sample before processing | Consistent per-request; simple to implement |
| **Tail-based** | Decide to sample after processing | Keep all errors/slow requests, drop routine logs |
| **Level-based** | Sample by severity | Always keep Warning+, sample Debug/Info |
| **Dynamic** | Adjust rate based on volume | Handle traffic spikes without config changes |

### OTel Collector Log Filtering

The `filter` processor in the OTel Collector drops log records at the pipeline level before they reach exporters. Use it to exclude noisy low-severity logs and reduce storage volume.

Note: The `tail_sampling` processor operates on **traces** (spans), not logs. For log volume management, use the `filter` and `transform` processors instead.

```yaml
processors:
  filter:
    logs:
      exclude:
        match_type: regexp
        # Drop Debug and Trace logs at the collector level
        severity_texts: ["DEBUG", "TRACE"]
      exclude:
        match_type: strict
        # Exclude health check noise
        bodies:
          - "Health check endpoint hit"
  transform:
    log_statements:
      - context: log
        conditions:
          # Keep all Warning+ logs unconditionally
          - severity_number >= SEVERITY_NUMBER_WARN
        statements: []
```

### Application-Level Sampling with Serilog

```csharp
// Serilog.Expressions package for conditional log filtering
builder.Host.UseSerilog((context, loggerConfiguration) =>
{
    loggerConfiguration
        .ReadFrom.Configuration(context.Configuration)
        // Drop health check logs entirely
        .Filter.ByExcluding("RequestPath = '/health/ready'")
        // Sample Debug logs at 10%
        .Filter.ByExcluding(
            "@Level = 'Debug' and Hash(@i) % 10 != 0");
});
```

**Key packages:**

```xml
<PackageReference Include="Serilog.Expressions" Version="5.*" />
```

### Volume Management Checklist

1. **Set retention policies** per index/stream (e.g., 30 days for Info, 90 days for Error)
2. **Use log level filtering** to suppress noisy framework categories at the source
3. **Exclude health check endpoints** from request logging
4. **Apply index lifecycle management** (ILM in Elasticsear