dt-obs-kubernetes

What this skill does

# Infrastructure Kubernetes

Monitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query
cluster resources, monitor workload health, analyze pod placement, optimize
costs, and assess security posture.

## When to Use This Skill

- Monitoring Kubernetes cluster health and capacity
- Analyzing pod and container resource utilization
- Investigating pod failures, OOMKills, evictions, or crash loops
- Debugging degraded deployments, stuck rollouts, or node pressure
- Optimizing Kubernetes resource costs
- Assessing security posture and compliance
- Troubleshooting workload scheduling and placement
- Auditing ingress routing and network policies
## Knowledge Base Structure

### Core Monitoring (Start Here)

1. **Cluster Inventory** → `references/cluster-inventory.md` - Clusters,
   namespaces, resource distribution
2. **Node Monitoring** - Node capacity, CPU/memory usage, pod density
3. **Pod Monitoring** - Pod CPU, memory, lifecycle events
4. **Workload Monitoring** - Deployment, StatefulSet, DaemonSet resources

### Advanced Topics

1. **Configuration Analysis** → `references/labels-annotations.md` - Parse
   k8s.object, labels, annotations
2. **Scheduling & Placement** → `references/pod-node-placement.md` - Node
   selectors, affinity, taints, HA
3. **Cost Optimization** - Right-sizing, waste detection, efficiency scoring
4. **Security & Compliance** - Privileged containers, security contexts

## Key Concepts

### Entity Types

**Workloads:** `K8S_DEPLOYMENT`, `K8S_STATEFULSET`, `K8S_DAEMONSET`,
`K8S_JOB`, `K8S_CRONJOB`, `K8S_HORIZONTALPODAUTOSCALER`  
**Infrastructure:** `K8S_CLUSTER`, `K8S_NAMESPACE`, `K8S_NODE`, `K8S_POD`  
**Configuration:** `K8S_SERVICE`, `K8S_CONFIGMAP`, `K8S_SECRET`,
`K8S_PERSISTENTVOLUMECLAIM`, `K8S_PERSISTENTVOLUME`, `K8S_INGRESS`,
`K8S_NETWORKPOLICY`

### Query Types

**smartscapeNodes** - Query K8s entities:

```dql
smartscapeNodes K8S_POD
| filter k8s.namespace.name == "production"
| fields k8s.cluster.name, k8s.pod.name
```

**timeseries** - Monitor metrics over time:

```dql
timeseries cpu = sum(dt.kubernetes.container.cpu_usage),
  by: {k8s.pod.name, k8s.namespace.name}
| fieldsAdd avg_cpu = arrayAvg(cpu)
```

**fetch logs** - Analyze log events:

```dql
fetch logs
| filter k8s.namespace.name == "production" and loglevel == "ERROR"
```

### Core Fields

- `k8s.cluster.name`, `k8s.namespace.name`, `k8s.pod.name`, `k8s.node.name`
- `k8s.workload.name`, `k8s.workload.kind`, `k8s.container.name`
- `k8s.object` - Full JSON configuration for deep inspection
- `tags[label]` - Access labels and annotations

### Available Metrics

**CPU:** `dt.kubernetes.container.cpu_usage`, `cpu_throttled`, `limits_cpu`,
`requests_cpu`  
**Memory:** `dt.kubernetes.container.memory_working_set`, `limits_memory`,
`requests_memory`  
**Operations:** `dt.kubernetes.container.restarts`, `oom_kills`  
**Node:** `dt.kubernetes.node.pods_allocatable`, `cpu_allocatable`,
`memory_allocatable`, `dt.kubernetes.pods`

### Entity Disambiguation

`K8S_POD` vs `CONTAINER`: these are different entity types in Dynatrace.

- **`K8S_POD`** — K8s-native entities with `k8s.object` JSON, scheduling state, conditions, and K8s metrics. Use this skill.
- **`CONTAINER`** — Host-level container inventory (image, lifetime, host assignment). Use `dt-obs-hosts` skill instead.

The smartscape edge is `CONTAINER --(is_part_of)--> K8S_POD`. To reach containers from a pod, traverse backward:

```dql-template
smartscapeNodes K8S_POD
| filter k8s.namespace.name == "<namespace>"
| traverse edgeTypes: {is_part_of}, targetTypes: {CONTAINER}, direction: backward, fieldsKeep: {id}
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, container.id=id
```

### Service → K8S_POD Correlation

No direct smartscape edge exists between `SERVICE` and `K8S_POD`. The correlation key is the shared dimension `k8s.workload.name`. See [Service → Pod Drill-Down](references/pod-debugging.md#service--pod-drill-down) in `references/pod-debugging.md` for the full two-step pattern.

## Common Workflows

### 1. Cluster Health Check

List all clusters:

```dql
smartscapeNodes K8S_CLUSTER
| fields k8s.cluster.name, k8s.cluster.version, k8s.cluster.distribution
```

Check node capacity:

```dql
timeseries {
  current_pods = avg(dt.kubernetes.pods),
  max_pods = avg(dt.kubernetes.node.pods_allocatable)
}, by: {k8s.node.name, k8s.cluster.name}
| fieldsAdd pod_capacity_pct = (arrayAvg(current_pods) / arrayAvg(max_pods)) * 100
| filter pod_capacity_pct > 80
```

Identify pods in non-Running state:

```dql
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| fieldsAdd phase = config[status][phase]
| filter phase != "Running"
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, phase
```

### 2. Resource Optimization

Find over-provisioned pods (usage < 30%):

```dql
timeseries {
  cpu_usage = sum(dt.kubernetes.container.cpu_usage),
  cpu_requests = avg(dt.kubernetes.container.requests_cpu)
}, by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name}
| fieldsAdd usage_pct = (arrayAvg(cpu_usage) / arrayAvg(cpu_requests)) * 100
| filter usage_pct < 30 and arrayAvg(cpu_requests) > 0
```

Identify containers without limits:

```dql
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| expand container = config[spec][containers]
| fieldsAdd
    container_name = container[name],
    cpu_limit = container[resources][limits][cpu],
    memory_limit = container[resources][limits][memory]
| filter isNull(cpu_limit) or isNull(memory_limit)
```

### 3. Troubleshooting Pod Issues

Pod troubleshooting benefits from combining **metrics** (timeseries) with
**Kubernetes events** (event stream) for a complete picture.

#### Metrics-Based Troubleshooting

Find pods with OOMKills:

```dql
timeseries oom_kills = sum(dt.kubernetes.container.oom_kills),
  by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name}
| filter arraySum(oom_kills) > 0
| fieldsAdd total_oom_kills = arraySum(oom_kills)
| sort total_oom_kills desc
```

Analyze pod restart patterns:

```dql
timeseries restarts = sum(dt.kubernetes.container.restarts),
  by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name}
| fieldsAdd total_restarts = arraySum(restarts)
| filter total_restarts > 5
```

#### Event-Based Troubleshooting

For operational events (pod restarts, OOM kills, evictions, scheduling failures),
Kubernetes events provide richer context than metrics alone — including event
reasons, messages, and timestamps.

**When to use Kubernetes events over metrics:**
- User asks about recent operational events ("show me pod restart events")
- User wants event details like reasons and messages
- User asks about events in a specific time window ("last 48 hours")
- User wants to correlate events with root causes

**Kubernetes events** are available through the `get-events-for-kubernetes-cluster`
tool. **Prefer this tool** when the user asks about OOM events, pod restarts,
evictions, or cluster-wide event history.

**Important: distinguish event types when filtering results.** Kubernetes events
cover many categories. When the user asks about a specific event type, filter
the results accordingly — do not report unrelated events:

| User Asks About | Relevant Event Reasons | NOT Related |
|-----------------|----------------------|-------------|
| Pod restarts | `BackOff`, `CrashLoopBackOff`, `Killing` | Readiness probe failures, CPU throttling |
| OOM events | `OOMKilling`, `OOMKilled` | Memory pressure warnings |
| Evictions | `Evicted`, `Preempting` | Node pressure |
| Scheduling failures | `FailedScheduling`, `Unschedulable` | Resource quotas |

**For a complete answer**, combine both approaches:
1. Use the **events tool** to get the event details (what happened, when, why)
2. Use **timeseries metrics** to show the quantitative impact (how many restarts,
   OOM kill counts over time)

#### Fetch Kubernetes Events via DQL

Pod restart and operational events can also be queried via DQL from the events
table:

```dql
fetch events
| filter