otel-observability

What this skill does

# OpenTelemetry Observability Agent

## What's New in OpenTelemetry (2024-2025)

- **Stable Logs**: Logging API and SDK now stable in many languages
- **Events API**: New semantic event support
- **Profiling signal**: CPU/memory profiling support (experimental)
- **Enhanced semantic conventions**: Standardized attribute names
- **Collector improvements**: Better performance and reliability
- **OTLP/JSON**: JSON encoding for OTLP widely supported

## Core Concepts

OpenTelemetry (OTel) provides three observability signals:

| Signal      | Purpose                      | Use Case                         |
| ----------- | ---------------------------- | -------------------------------- |
| **Traces**  | Request flow across services | Debugging distributed systems    |
| **Metrics** | Numerical measurements       | Performance monitoring, alerting |
| **Logs**    | Structured event records     | Error tracking, audit trails     |
| **Baggage** | Context propagation          | Passing data across services     |

## Installation

### Node.js Packages

```bash
# Core packages
npm install @opentelemetry/api
npm install @opentelemetry/sdk-node
npm install @opentelemetry/sdk-trace-node
npm install @opentelemetry/sdk-metrics

# Auto-instrumentation
npm install @opentelemetry/auto-instrumentations-node

# OTLP exporters
npm install @opentelemetry/exporter-trace-otlp-http
npm install @opentelemetry/exporter-metrics-otlp-http
```

## Zero-Code Instrumentation

### Environment Variables

```bash
# Run with auto-instrumentation
OTEL_TRACES_EXPORTER="otlp" \
OTEL_METRICS_EXPORTER="otlp" \
OTEL_LOGS_EXPORTER="otlp" \
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" \
OTEL_SERVICE_NAME="my-service" \
OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=production" \
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
node app.js
```

### Common Environment Variables

| Variable                      | Description        | Example                   |
| ----------------------------- | ------------------ | ------------------------- |
| `OTEL_SERVICE_NAME`           | Service identifier | `"user-service"`          |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector endpoint | `"http://localhost:4318"` |
| `OTEL_TRACES_EXPORTER`        | Trace exporter     | `"otlp"`, `"console"`     |
| `OTEL_METRICS_EXPORTER`       | Metrics exporter   | `"otlp"`, `"prometheus"`  |
| `OTEL_LOGS_EXPORTER`          | Logs exporter      | `"otlp"`, `"console"`     |
| `OTEL_TRACES_SAMPLER`         | Sampling strategy  | `"parentbased_always_on"` |
| `OTEL_TRACES_SAMPLER_ARG`     | Sampler argument   | `"0.1"` (10% sampling)    |

## Programmatic Setup

### Basic Node.js SDK

```typescript
// instrumentation.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";

const sdk = new NodeSDK({
  serviceName: "my-service",

  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4318/v1/traces",
  }),

  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: "http://localhost:4318/v1/metrics",
    }),
    exportIntervalMillis: 60000,
  }),

  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Graceful shutdown
process.on("SIGTERM", () => {
  sdk.shutdown().then(() => process.exit(0));
});
```

### Import Early

```typescript
// app.ts - instrumentation MUST be imported first
import "./instrumentation";

import express from "express";
// ... rest of app
```

## Tracing

### Getting a Tracer

```typescript
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("my-service", "1.0.0");
```

### Creating Spans

```typescript
// Automatic span management (recommended)
tracer.startActiveSpan("operation-name", (span) => {
  try {
    // Your code here
    span.setAttribute("user.id", userId);
    return result;
  } catch (error) {
    span.recordException(error as Error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
});

// Async operations
async function processOrder(orderId: string) {
  return tracer.startActiveSpan("process-order", async (span) => {
    try {
      span.setAttribute("order.id", orderId);
      const result = await orderService.process(orderId);
      return result;
    } finally {
      span.end();
    }
  });
}
```

### Span Kinds

```typescript
import { SpanKind } from "@opentelemetry/api";

// CLIENT - outgoing request (HTTP client, DB call)
tracer.startActiveSpan(
  "fetch-user",
  { kind: SpanKind.CLIENT },
  async (span) => {
    const user = await fetch("/api/users/1");
    span.end();
  },
);

// SERVER - incoming request (HTTP handler)
tracer.startActiveSpan("handle-request", { kind: SpanKind.SERVER }, (span) => {
  // Handle incoming HTTP request
  span.end();
});

// PRODUCER - message production
tracer.startActiveSpan("send-message", { kind: SpanKind.PRODUCER }, (span) => {
  queue.send(message);
  span.end();
});

// CONSUMER - message consumption
tracer.startActiveSpan(
  "process-message",
  { kind: SpanKind.CONSUMER },
  (span) => {
    processMessage(message);
    span.end();
  },
);

// INTERNAL - internal operation (default)
tracer.startActiveSpan("calculate", { kind: SpanKind.INTERNAL }, (span) => {
  const result = heavyCalculation();
  span.end();
});
```

### Span Attributes

```typescript
import { SpanStatusCode } from "@opentelemetry/api";

tracer.startActiveSpan("http-request", (span) => {
  // Set attributes
  span.setAttribute("http.method", "GET");
  span.setAttribute("http.url", "https://api.example.com/users");
  span.setAttribute("http.status_code", 200);

  // Set multiple attributes
  span.setAttributes({
    "user.id": "123",
    "user.role": "admin",
    "request.cached": false,
  });

  // Add events
  span.addEvent("cache-miss", {
    "cache.key": "user:123",
  });

  // Set status
  span.setStatus({ code: SpanStatusCode.OK });

  span.end();
});
```

### Error Handling

```typescript
tracer.startActiveSpan("risky-operation", (span) => {
  try {
    riskyOperation();
  } catch (error) {
    // Record the exception
    span.recordException(error as Error);

    // Set error status
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: (error as Error).message,
    });

    throw error;
  } finally {
    span.end();
  }
});
```

## Metrics

### Getting a Meter

```typescript
import { metrics } from "@opentelemetry/api";

const meter = metrics.getMeter("my-service", "1.0.0");
```

### Counter (Monotonic Increasing)

```typescript
// Create counter
const requestCounter = meter.createCounter("http.requests.total", {
  description: "Total number of HTTP requests",
  unit: "1",
});

// Increment
requestCounter.add(1, {
  "http.method": "GET",
  "http.route": "/api/users",
  "http.status_code": 200,
});
```

### UpDownCounter (Can Decrease)

```typescript
const activeConnections = meter.createUpDownCounter("connections.active", {
  description: "Number of active connections",
  unit: "1",
});

// Increment on connect
activeConnections.add(1);

// Decrement on disconnect
activeConnections.add(-1);
```

### Histogram (Distribution)

```typescript
const requestDuration = meter.createHistogram("http.request.duration", {
  description: "HTTP request duration",
  unit: "ms",
});

// Record value
const start = performance.now();
await handleRequest();
const duration = performance.now() - start;

requestDuration.record(duration, {
  "http.method": "POST",
  "http.route": "/api/orders",
});
```

### Observable Gauge (Async Measurement)

```typescript
// For values that are measured periodically
const memoryUsage = meter.createObservableGaug