otel-tracing

What this skill does

# OTel Tracing

Create spans for logical operations, propagate context across every boundary, use semantic conventions for attribute
names. Every tracing decision — span granularity, attribute selection, sampling strategy — trades off cost against
visibility.

## References

- **Spans** — [`${CLAUDE_SKILL_DIR}/references/spans.md`]: Span anatomy, root spans, lifetime code patterns
- **Span data** — [`${CLAUDE_SKILL_DIR}/references/span-data.md`]: Events format, links format, SDK limits table
- **Context propagation** — [`${CLAUDE_SKILL_DIR}/references/context-propagation.md`]: W3C header format, propagator
  selection, baggage details, security
- **Instrumentation** — [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`]: Server/client/async code patterns,
  library rules, testing guidance
- **Sampling** — [`${CLAUDE_SKILL_DIR}/references/sampling.md`]: Head/tail/combined strategies, decision guide, sampler
  types
- **Semantic conventions** — [`${CLAUDE_SKILL_DIR}/references/semantic-conventions.md`]: HTTP/DB/messaging attribute
  lists, status mapping, general conventions
- **SDK components** — [`${CLAUDE_SKILL_DIR}/references/sdk-components.md`]: Resource config, env vars, Collector
  deployment, exporter types

## Spans

A span represents one unit of work in a trace — an HTTP request handler, a database query, a message publish. Not every
function call.

### Span Naming

Name spans for the **class** of operation, not the instance. Low-cardinality names enable aggregation; high-cardinality
names destroy it.

- Use the most general string that identifies a statistically interesting class
- Never embed high-cardinality values (IDs, emails, paths with IDs)
- HTTP: `{METHOD} {route}` — server: `GET /users/:id`; client: `GET`
- DB: `{operation} {table}` — `SELECT users`, `INSERT orders`

| Name                         | Verdict                   |
| ---------------------------- | ------------------------- |
| `get`                        | Too general               |
| `get_account/42`             | Too specific — ID in name |
| `get_account`                | Good                      |
| `GET /users/{userId}`        | Good — route template     |
| `POST /api/v2/orders/abc123` | Bad — order ID in name    |

### SpanKind

SpanKind tells backends how to assemble the trace tree. Set it correctly.

- `SERVER` (Incoming) — HTTP handler, gRPC server method
- `CLIENT` (Outgoing) — HTTP client call, DB query, gRPC call
- `PRODUCER` (Outgoing) — Enqueue message, schedule job
- `CONSUMER` (Incoming) — Dequeue message, process job
- `INTERNAL` (Neither) — In-process business logic, computation

- A single span MUST NOT serve more than one purpose
- Create a new span before injecting context for outgoing calls
- `CLIENT` -> `SERVER` for synchronous calls; `PRODUCER` -> `CONSUMER` for async
- Using `INTERNAL` for HTTP handlers or DB calls is wrong — use `SERVER` or `CLIENT`

### Span Status

- `Unset` — No error. Default — do not change on success.
- `Error` — Operation failed. When an error occurs; include description.
- `Ok` — Explicitly successful. Only to override a previous `Error`; rarely needed.

- Leave status `Unset` on success — it already means "no error"
- `Ok` is a "final call" — once set, subsequent `Error` attempts are ignored
- Instrumentation libraries SHOULD NOT set `Ok` — leave to application code

### Span Lifetime

- **Every created span MUST be ended** — leaked spans cause memory issues and incomplete traces
- After `end()`, the span becomes non-recording; further mutations are ignored
- Use language-idiomatic patterns: `defer` (Go), try/finally (Java/JS), context manager (Python)

## Span Data: Attributes, Events, Links

### Attributes

Key-value pairs annotating a span. Value types: string, boolean, integer, float, or arrays of these.

- **Use semantic conventions** for attribute names — `http.request.method`, `db.system`, not custom names. Consistent
  naming enables cross-service analysis.
- **Add sampling-relevant attributes at span creation** — samplers can only see attributes present at creation time.
- **Keep cardinality low** — attribute values should be from a bounded set.
- **Check `IsRecording` before expensive attribute computation** — sampled-out spans discard all data.
- **Never store PII in attributes** — trace data flows to shared backends. Use opaque identifiers.
- **Respect attribute limits** — SDK enforces `AttributeCountLimit` (default 128). Excess attributes are silently
  dropped.

### Events

Timestamped annotations — structured log entries attached to a span.

- Use events when the timestamp matters; use attributes for metadata with no meaningful timestamp
- Add events for key domain occurrences ("page became interactive", "retry attempted")
- Use events for verbose data instead of additional spans

### Recording Exceptions

- **Always pair `RecordException` with `setStatus(ERROR)`** — RecordException alone does not change span status
- Never swallow application exceptions in instrumentation — catch instrumentation errors, log them, but always rethrow
  application exceptions

### Links

Associate a span with spans from other traces without parent-child hierarchy.

- **Batch processing:** link consumer span to each producer span
- **Async follow-up:** job span links back to triggering span
- Add links at span creation when possible — samplers can consider them

See `${CLAUDE_SKILL_DIR}/references/span-data.md` for events format, links format, and SDK limits table.

## Context Propagation

Context propagation correlates spans across service boundaries. Without it, each service produces isolated spans — no
trace.

### Inject / Extract Pattern

1. **Sender** creates a span, makes it current, **injects** context into outgoing carrier (HTTP headers, message
   metadata)
2. **Receiver** **extracts** context from incoming carrier and uses it as parent for new spans

- Call inject AFTER creating the outgoing CLIENT or PRODUCER span
- Call extract BEFORE creating the server/consumer span
- Use W3C TraceContext as the default propagation format

### In-Process Propagation

- **Make spans active/current** — enables automatic log correlation, nested auto-instrumentation picking up correct
  parent, context propagation to child ops
- **Pass context explicitly in async code** — automatic propagation may break across thread boundaries, goroutines, or
  async callbacks
- **Never propagate request context to background work** — background tasks should start new traces and link back to the
  triggering span

### Baggage

Propagates arbitrary key-value pairs across service boundaries alongside trace context.

- Use for: tenant ID, request priority, feature flags, sampling hints
- **NEVER put PII, credentials, or API keys in baggage** — visible to all downstream services

See `${CLAUDE_SKILL_DIR}/references/context-propagation.md` for W3C header format, propagator selection table, baggage
details, and security considerations.

## Instrumentation

### Approaches

| Approach             | When                                      | Trade-off                                  |
| -------------------- | ----------------------------------------- | ------------------------------------------ |
| Automatic            | Supported libraries (HTTP, DB, messaging) | Zero code, no business attrs               |
| Manual               | Business logic, unsupported libraries     | Full control, more code                    |
| Hybrid (recommended) | Production                                | Auto for infra + manual for business logic |

### Getting a Tracer

- Name the tracer after your library, package, or module — not the application
- Include a version string matching your library version
- Libraries: accept `TracerProvider` via dependency injection or use the global one
- Applications: configure the SDK and set the global `TracerProvider`

### What to Instrument

**Good candidates:** public API methods with I/O or significant computation, request