dt-obs-services

What this skill does

# Application Services Skill

Monitor application service performance, health, and runtime-specific metrics using DQL.

---

## Core Capabilities

### 1. Service Performance (RED Metrics)

Monitor service **Rate, Errors, Duration** using metrics-based timeseries queries.

**Key Metrics:**
- `dt.service.request.response_time` - Response time (microseconds)
- `dt.service.request.count` - Request count
- `dt.service.request.failure_count` - Failed request count

**Common Use Cases:**
- Response time monitoring (avg, p50, p95, p99)
- Error rate tracking and spike detection
- Traffic analysis (throughput, peaks, growth)
- Performance degradation detection
- Multi-cluster comparison

**Quick Example:**
```dql
timeseries {
  p95 = percentile(dt.service.request.response_time, 95),
  total_requests = sum(dt.service.request.count),
  failures = sum(dt.service.request.failure_count)
}, by: {dt.service.name}
| fieldsAdd p95_ms = p95[] / 1000, error_rate_pct = (failures[] * 100.0) / total_requests[]
```

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

### 2. Advanced Service Analysis

Span-based queries for complex scenarios requiring flexible filtering and custom aggregations.

**Use Cases:**
- SLA compliance tracking with custom thresholds
- Service health scoring (multi-dimensional)
- Operation/endpoint-level performance analysis
- Custom error classification
- Failure pattern detection with error details

**Quick Example:**
```dql
fetch spans, from: now() - 1h | filter request.is_root_span == true
| fieldsAdd meets_sla = if(request.is_failed == false AND duration < 3s, 1, else: 0)
| summarize total = count(), sla_compliant = sum(meets_sla), by: {dt.service.name}
| fieldsAdd sla_compliance_pct = (sla_compliant * 100.0) / total
```

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

### 3. Service Messaging Metrics

Monitor message-based service communication (queues, topics).

**Key Metrics:**
- `dt.service.messaging.publish.count` - Messages sent to queues or topics
- `dt.service.messaging.receive.count` - Messages received from queues or topics
- `dt.service.messaging.process.count` - Messages successfully processed
- `dt.service.messaging.process.failure_count` - Messages that failed processing

**Use Cases:**
- Message throughput monitoring (publish/receive rates)
- Message processing failure tracking
- Queue/topic health analysis
- Consumer lag detection (publish vs receive rate comparison)

**Quick Example:**
```dql
timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count),
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}
```

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

### 4. Service Mesh Monitoring

Monitor service mesh ingress performance and overhead.

**Key Metrics:**
- `dt.service.request.service_mesh.response_time` - Mesh response time (microseconds)
- `dt.service.request.service_mesh.count` - Mesh request count
- `dt.service.request.service_mesh.failure_count` - Mesh failure count

**Use Cases:**
- Mesh vs direct performance comparison
- Mesh overhead calculation
- Mesh failure analysis
- gRPC traffic monitoring
- Multi-cluster mesh performance

**Quick Example:**
```dql
timeseries {
  direct_p95 = percentile(dt.service.request.response_time, 95),
  mesh_p95 = percentile(dt.service.request.service_mesh.response_time, 95)
}, by: {dt.service.name}
| fieldsAdd mesh_overhead_ms = (mesh_p95[] - direct_p95[]) / 1000
```

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

### 5. Runtime-Specific Monitoring

Technology-specific runtime performance and resource usage metrics.

**Java/JVM** - [references/java.md](references/java.md)
- Memory: heap, pools, metaspace
- GC: impact, suspension, frequency, pause time
- Threads: count monitoring, leak detection
- Classes: loading, unloading, growth

**Node.js** - [references/nodejs.md](references/nodejs.md)
- Event loop: utilization, active handles
- V8 heap: memory used, total
- GC: collection time, suspension
- Process: RSS memory

**.NET CLR** - [references/dotnet.md](references/dotnet.md)
- Memory: consumption by generation
- GC: collection count, suspension time
- Thread pool: threads, queued work
- JIT: compilation time

**Python** - [references/python.md](references/python.md)
- Threads: active thread count
- Heap: allocated blocks
- GC: collection by generation, pause time
- Objects: collected, uncollectable

**PHP** - [references/php.md](references/php.md)
- OPcache: hit ratio, memory, restarts
- GC: effectiveness, duration
- JIT: buffer usage
- Interned strings: usage, buffer

**Go** - [references/go.md](references/go.md)
- Goroutines: count, leak detection
- GC: suspension, collection time
- Memory: heap by state, committed
- Scheduler: worker threads, queue size
- CGo: call frequency

---

## When to Use This Skill

✅ **Use for:**
- Monitoring service performance (response time, errors, traffic)
- Calculating SLA compliance
- Analyzing service mesh performance
- Monitoring messaging throughput and processing failures
- Troubleshooting runtime-specific issues (GC, memory, threads)
- Multi-cluster service comparison
- Operation/endpoint-level analysis

❌ **Don't use for:**
- Infrastructure metrics (use infrastructure skills)
- Log analysis (use logs skills)
- Distributed tracing workflows (use traces/spans skills)
- Database performance (use database skills)
- Product documentation or how-to configuration questions → use `ask-dynatrace-docs`

---

## Agent Instructions

### Act First, Refine Later

When a user asks for analysis — threshold checks, anomaly detection, performance
comparisons — **proceed immediately** with sensible defaults. Do not ask the user
for parameter values you can reasonably assume.

Why this matters: analysis tools (e.g., `static-threshold-analyzer`) require specific
inputs like threshold values and service scope. The user expects results, not a
parameter interview. Pick reasonable defaults, state them clearly in the response,
and let the user refine.

**Default values when not specified:**

| Parameter | Default | Rationale |
|-----------|---------|-----------|
| Response time threshold | 1000 ms (= 1,000,000 µs in the metric's base unit) | Common SLA boundary |
| Service scope | All services | Show the most relevant violations |
| Timeframe | From the request, or last 30 min for threshold checks, 2h for general analysis | Matches typical operational windows |

**Example: threshold violation request**
1. Use `create-dql` to build a timeseries query for `avg(dt.service.request.response_time)` grouped by `dt.smartscape.service`
2. Pass the query to `static-threshold-analyzer` with threshold = 1000000 (µs), alertCondition = ABOVE
3. Resolve entity IDs to names using `get-entity-name`
4. Present violations with service names, timestamps, values, and duration

**Reading user phrasing:** Phrases like "the fixed threshold", "a threshold", or "the limit"
name the *type of analysis* — static threshold check — not a specific number the user expects
you to already know. "Fixed" distinguishes a static cutoff from a dynamic or seasonal baseline.
When you see these phrases, apply the 1000 ms default from the table above and present
results — the user can then refine if the default doesn't match their intent.

### Scope Boundary

This skill covers **service performance metrics and runtime monitoring only**. If the
user asks a product documentation or configuration question (e.g., "How do I add custom
sensors?", "How do I configure service detection?"), use `ask-dynatrace-docs` instead —
this skill does not contain configuration how-tos.

### Understanding User Intent

**Map user questions to capabilities:**

| User Request | Use Capability | Key Files |