otel-logging-patterns

What this skill does

# OpenTelemetry Logging Patterns

## Table of Contents

- [Purpose](#purpose)
- [Quick Start](#quick-start)
- [Instructions](#instructions)
  - [Step 1: Configure OTEL Logging Provider](#step-1-configure-otel-logging-provider)
  - [Step 2: Integrate Structured Logging Library](#step-2-integrate-structured-logging-library)
  - [Step 3: Add Trace Context Propagation](#step-3-add-trace-context-propagation)
  - [Step 4: Set Up Log Exporters](#step-4-set-up-log-exporters)
  - [Step 5: Implement Instrumentation](#step-5-implement-instrumentation)
  - [Step 6: Add Error and Exception Logging](#step-6-add-error-and-exception-logging)
- [Requirements](#requirements)
- [Common Patterns](#common-patterns)
- [Testing OTEL Logging](#testing-otel-logging)
- [Troubleshooting](#troubleshooting)
- [Supporting Resources](#supporting-resources)

## Purpose

This skill provides production-grade OpenTelemetry logging patterns for Python applications. It covers:

- **OTEL Logging Architecture**: Provider and processor configuration
- **Trace Correlation**: Automatic trace context injection into logs
- **Structured Logging**: Integration with structlog for context preservation
- **Log Exporters**: OTLP, Jaeger, console, and file exporters
- **Error Handling**: Comprehensive exception and error logging
- **Testing Patterns**: Unit and integration tests for logging infrastructure
- **Performance**: Optimization tips to minimize logging overhead

This enables production observability where logs, traces, and metrics are correlated through trace IDs for efficient debugging and monitoring.

## Quick Start

**For this project**, use the authoritative OTEL logging module. Get started in 3 simple steps:

1. **Initialize OTEL at application startup** (once):

```python
from app.core.monitoring.otel_logger import initialize_otel_logger

# Call once at app startup
initialize_otel_logger(
    log_level="INFO",
    enable_console=True,
    enable_otlp=True,
    otlp_endpoint="localhost:4317"
)
```

2. **Get logger and tracer in each module**:

```python
from app.core.monitoring.otel_logger import logger, get_tracer

# Module-level initialization
logger = logger(__name__)
tracer = get_tracer(__name__)
```

3. **Use trace_span for operations**:

```python
from app.core.monitoring.otel_logger import trace_span, logger

logger = logger(__name__)

# Logs automatically include trace_id and span_id
with trace_span("process_order", order_id="12345") as span:
    logger.info("processing_order", order_id="12345")
    # Do work...
    logger.info("order_processed", result_count=5)
```

That's it! All logs automatically include trace context, and spans are created with attributes.

**Key Benefits**:
- ✅ Single entry point: `app/core/monitoring/otel_logger.py`
- ✅ No direct imports of structlog or opentelemetry needed
- ✅ Automatic trace context propagation
- ✅ Automatic exception handling in spans
- ✅ Works with async and sync functions

## Instructions

### Step 1: Configure OTEL Logging Provider

Set up the core OpenTelemetry logging infrastructure with provider and exporter configuration.

**Basic setup** (console exporter for development):

```python
# app/shared/otel_config.py
from opentelemetry import logs
from opentelemetry.sdk.logs import LoggerProvider
from opentelemetry.sdk.logs.export import ConsoleLogExporter, SimpleLogRecordExporter
from opentelemetry.sdk.resources import Resource

def setup_console_logging(service_name: str) -> LoggerProvider:
    """Set up OTEL logging with console exporter."""
    resource = Resource.create({"service.name": service_name})
    logger_provider = LoggerProvider(resource=resource)

    exporter = ConsoleLogExporter()
    processor = SimpleLogRecordExporter(exporter)
    logger_provider.add_log_record_processor(processor)
    logs.set_logger_provider(logger_provider)

    return logger_provider
```

**For production OTLP/Jaeger setup**, see [references/advanced-patterns.md](references/advanced-patterns.md#otelconfig-class)

### Step 2: Integrate Structured Logging Library

Set up structlog with OTEL trace context integration:

```python
# app/shared/logging_setup.py
import structlog
from opentelemetry.instrumentation.logging import LoggingInstrumentor

def setup_structlog() -> None:
    """Configure structlog with OTEL trace context integration."""
    # Enable OTEL logging instrumentation
    LoggingInstrumentor().instrument()

    # Configure structlog
    structlog.configure(
        processors=[
            structlog.contextvars.merge_contextvars,
            structlog.processors.TimeStamper(fmt="iso"),
            structlog.processors.ExceptionRenderer(),
            structlog.processors.JSONRenderer(),
        ],
        logger_factory=structlog.logging.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

def get_logger(name: str):
    """Get a logger instance with trace context support."""
    return structlog.logger(name)
```

**For advanced structlog configuration**, see [references/advanced-patterns.md](references/advanced-patterns.md#structlog-configuration)

### Step 3: Add Trace Context Propagation

Ensure trace context flows through logs automatically:

```python
# app/shared/observability.py
from contextvars import ContextVar
from opentelemetry import trace

# Context variables for request tracking
request_id_var: ContextVar[str | None] = ContextVar("request_id", default=None)
user_id_var: ContextVar[str | None] = ContextVar("user_id", default=None)

class ObservabilityContext:
    """Manage observability context (trace IDs, request IDs, user IDs)."""

    @staticmethod
    def set_request_id(request_id: str) -> None:
        """Set request ID in context."""
        request_id_var.set(request_id)

    @staticmethod
    def get_tracer(name: str):
        """Get tracer instance."""
        return trace.get_tracer(name)

    @staticmethod
    def set_span_attribute(key: str, value: any) -> None:
        """Set attribute on current span."""
        span = trace.get_current_span()
        if span.is_recording():
            span.set_attribute(key, value)
```

**For complete ObservabilityContext class**, see [references/advanced-patterns.md](references/advanced-patterns.md#observability-context)

### Step 4: Set Up Log Exporters

Configure different exporters for different environments:

```python
# app/config.py
from pydantic import Field
from pydantic_settings import BaseSettings

class Config(BaseSettings):
    """Configuration for OTEL logging."""

    otel_enabled: bool = Field(default=True)
    otel_exporter_type: str = Field(default="console")  # 'otlp', 'jaeger', or 'console'
    otel_otlp_endpoint: str = Field(default="localhost:4317")
    otel_jaeger_host: str = Field(default="localhost")
    otel_jaeger_port: int = Field(default=6831)
```

Then use config in main:

```python
# main.py
from app.config import Config
from app.shared.logging_setup import setup_structlog, get_logger
from app.shared.otel_config import OTELConfig

async def main() -> None:
    """Main entry point."""
    config = Config()
    setup_structlog()

    if config.otel_enabled:
        otel_config = OTELConfig(
            service_name="my-service",
            exporter_type=config.otel_exporter_type,
            otlp_endpoint=config.otel_otlp_endpoint,
        )
        otel_config.setup_logging()
        otel_config.setup_tracing()

    logger = get_logger(__name__)
    logger.info("service_started")
```

**For complete exporter configuration**, see [references/advanced-patterns.md](references/advanced-patterns.md#exporter-configuration)

### Step 5: Implement Instrumentation

Add tracing and logging to key application flows:

```python
# app/use_cases/extract_orders.py
from opentelemetry import trace
from app.shared.logging_setup import get_logger

class ExtractOrdersUseCase:
    """Use case with observability."""

    def __init__(self, gateway, publisher) -> None:
        self.gateway = gateway
        self.publisher = publisher
        self.log