dt-obs-hosts

What this skill does

# Infrastructure Hosts Skill

Monitor and manage host and process infrastructure including CPU, memory, disk, network, and technology inventory.

## When to Use This Skill

Use this skill when the user needs to:

- **Inventory:** "Show me all Linux hosts in AWS us-east-1"
- **Monitor:** "What hosts have high CPU usage?"
- **Troubleshoot:** "Which processes are consuming the most memory?"
- **Discover:** "What databases are running in production?"
- **Plan:** "Track Kubernetes version distribution for upgrade planning"
- **Cost:** "Calculate infrastructure costs by cost center"
- **Security:** "Find all processes listening on port 22"
- **Compliance:** "Identify hosts running EOL Java versions"
- **Quality:** "Check data completeness for AWS hosts"
- **Optimize:** "Find rightsizing candidates based on utilization"

---
> **Cross-source join required:** If the query must combine host data with logs or other
> telemetry sources (e.g. "show logs from Linux hosts with their IP addresses") → also read
> `dt-dql-essentials/references/smartscape-topology-navigation.md` before writing the query.
---

## Core Concepts

### Entities
- **HOST** - Physical or virtual machines (cloud or on-premise)
- **PROCESS** - Running processes and process groups
- **CONTAINER** - Kubernetes containers
- **NETWORK_INTERFACE** - Host network interfaces
- **DISK** - Host disk volumes

### Metrics Categories
1. **Host Metrics** - `dt.host.cpu.*`, `dt.host.memory.*`, `dt.host.disk.*`, `dt.host.net.*`
2. **Process Metrics** - `dt.process.cpu.*`, `dt.process.memory.*`, `dt.process.io.*`, `dt.process.network.*`
3. **Inventory** - OS type, cloud provider, technology stack, versions
4. **Cost** - `dt.cost.costcenter`, `dt.cost.product`
5. **Quality** - Metadata completeness, version compliance

### Alert Thresholds
- **CPU/Memory/Disk:** 80% warning, 90% critical
- **Network:** >70% high, >85% saturated
- **Disk Latency:** >20ms bottleneck
- **Network Errors:** Drop rate >1%, error rate >0.1%
- **Swap:** >30% warning, >50% critical

---

## Key Workflows

### 1. Host Discovery and Classification

Discover hosts, classify by OS/cloud, inventory resources.

```dql
smartscapeNodes "HOST"
| fieldsAdd os.type, cloud.provider, host.logical.cpu.cores, host.physical.memory
| summarize host_count = count(), by: {os.type, cloud.provider}
| sort host_count desc
```

**OS Types:** `LINUX`, `WINDOWS`, `AIX`, `SOLARIS`, `ZOS`

→ For cloud-specific attributes, see [references/inventory-discovery.md](#cloud-specific-attributes)

### 2. Resource Utilization Monitoring

Monitor CPU, memory, disk, network across hosts.

```dql
timeseries {
  cpu = avg(dt.host.cpu.usage),
  memory = avg(dt.host.memory.usage),
  disk = avg(dt.host.disk.used.percent)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(cpu) > 80 or arrayAvg(memory) > 80
| sort arrayAvg(cpu) desc
```

**High utilization threshold:** 80% warning, 90% critical

**Key CPU Metrics:**
- `dt.host.cpu.usage` — Total CPU utilization (0-100%)
- `dt.host.cpu.idle` — CPU idle time (inverse of usage; useful for anomaly detection)
- `dt.host.cpu.user` — CPU time in user mode
- `dt.host.cpu.system` — CPU time in kernel mode
- `dt.host.cpu.iowait` — CPU waiting for I/O (Linux only)

→ For detailed CPU analysis, see [references/host-metrics.md](references/host-metrics.md#cpu-monitoring)  
→ For memory breakdown, see [references/host-metrics.md](references/host-metrics.md#memory-monitoring)

#### Disk Free Space — Find Hosts with Most/Least Free Disk

```dql
timeseries disk_used_pct = avg(dt.host.disk.used.percent), by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd avg_disk_used = arrayAvg(disk_used_pct),
    free_pct = 100 - arrayAvg(disk_used_pct)
| sort free_pct desc
| limit 10
```

### 3. Process Resource Analysis

Identify top resource consumers at process level.

```dql
timeseries {
  cpu = avg(dt.process.cpu.usage),
  memory = avg(dt.process.memory.usage)
}, by: {dt.smartscape.process}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(cpu) > 50
| sort arrayAvg(cpu) desc
| limit 20
```

→ For process I/O analysis, see [references/process-monitoring.md](references/process-monitoring.md#process-io)  
→ For process network metrics, see [references/process-monitoring.md](references/process-monitoring.md#process-network)

### 4. Technology Stack Inventory

Discover and track software technologies and versions.

```dql
smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| expand tech = process.software_technologies
| fieldsAdd tech_type = tech[type], tech_version = tech[version]
| summarize process_count = count(), by: {tech_type, tech_version}
| sort process_count desc
```

**Common Technologies:** Java, Node.js, Python, .NET, databases, web servers, messaging systems

→ For version compliance checks, see [references/inventory-discovery.md](references/inventory-discovery.md#technology-inventory)

### 5. Service Discovery via Ports

Map listening ports to services for security and inventory.

```dql
smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports) and arraySize(process.listen_ports) > 0
| expand listen_port = process.listen_ports
| summarize process_count = count(), by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc
| limit 50
```

**Well-known ports:** 80 (HTTP), 443 (HTTPS), 22 (SSH), 3306 (MySQL), 5432 (PostgreSQL)

→ For comprehensive port mapping, see [references/inventory-discovery.md](references/inventory-discovery.md#port-discovery)

### 6. Container and Kubernetes Monitoring

Track container distribution and K8s workload types.

```dql
smartscapeNodes "CONTAINER"
| fieldsAdd k8s.cluster.name, k8s.namespace.name, k8s.workload.kind
| summarize container_count = count(), by: {k8s.cluster.name, k8s.workload.kind}
| sort k8s.cluster.name, container_count desc
```

**Workload Types:** `deployment`, `daemonset`, `statefulset`, `job`, `cronjob`

**Note:** Container image names/versions NOT available in smartscape.

→ For K8s version tracking, see [references/container-monitoring.md](references/container-monitoring.md#kubernetes-versions)  
→ For container lifecycle, see [references/container-monitoring.md](references/container-monitoring.md#container-inventory)

### 7. Cost Attribution and Chargeback

Calculate infrastructure costs by cost center.

```dql
smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, host.logical.cpu.cores, host.physical.memory
| filter isNotNull(dt.cost.costcenter)
| fieldsAdd memory_gb = toDouble(host.physical.memory) / 1024 / 1024 / 1024
| summarize 
    host_count = count(),
    total_cores = sum(toLong(host.logical.cpu.cores)),
    total_memory_gb = sum(memory_gb),
    by: {dt.cost.costcenter}
| sort total_cores desc
```

→ For product-level cost tracking, see [references/inventory-discovery.md](references/inventory-discovery.md#cost-attribution)

### 8. Infrastructure Health Correlation

Correlate host and process metrics for cross-layer analysis.

```dql
timeseries {
  host_cpu = avg(dt.host.cpu.usage),
  host_memory = avg(dt.host.memory.usage),
  process_cpu = avg(dt.process.cpu.usage)
}, by: {dt.smartscape.host, dt.smartscape.process}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(host_cpu) > 70
| sort arrayAvg(host_cpu) desc
```

**Health scoring:** Critical if any resource >90%, warning if >80%

→ For multi-resource saturation detection, see [references/host-metrics.md](references/host-metrics.md#resource-saturation)

---

## Response Construction

When the user asks for data retrieval or a DQL query (e.g., "show me top hosts by
CPU"), **include the DQL query in the response** alongside the results. Users want to
see and reuse the query — it is the deliverable, not just a means to get re