kubectl-debugging

What this skill does

# kubectl debug - Interactive Kubernetes Debugging

Expert knowledge for debugging Kubernetes resources using `kubectl debug` - ephemeral containers, pod copies, and node access.

## When to Use This Skill

| Use this skill when... | Use <sibling> instead when... |
|---|---|
| Attaching an ephemeral debug container to a running pod with `kubectl debug` | Use kubernetes-operations for general kubectl workflows (apply, get, describe, logs) |
| Creating a pod copy with a different image or command for interactive troubleshooting | Use helm-debugging when the failure is in template rendering or chart configuration, not the running container |
| Opening a node-level debug session to inspect host namespaces or filesystems | Use helm-release-recovery when the recovery action is a Helm rollback rather than per-pod debugging |

## Core Capabilities

**kubectl debug** automates common debugging tasks:

- **Ephemeral Containers**: Add debug containers to running pods without restart
- **Pod Copying**: Create modified copies for debugging (different images, commands)
- **Node Debugging**: Access node host namespaces and filesystem

## Context Safety (CRITICAL)

**Always specify `--context`** explicitly in every kubectl command:

```bash
# CORRECT: Explicit context
kubectl --context=prod-cluster debug mypod -it --image=busybox

# WRONG: Relying on current context
kubectl debug mypod -it --image=busybox  # Which cluster?
```

## Quick Reference

### Add Ephemeral Debug Container

```bash
# Interactive debugging with busybox
kubectl --context=my-context debug mypod -it --image=busybox

# Target specific container's process namespace
kubectl --context=my-context debug mypod -it --image=busybox --target=mycontainer

# Use a specific debug profile
kubectl --context=my-context debug mypod -it --image=busybox --profile=netadmin
```

### Copy Pod for Debugging

```bash
# Create debug copy
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --image=busybox

# Copy and change container image
kubectl --context=my-context debug mypod --copy-to=mypod-debug --set-image=app=busybox

# Copy and modify command
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --container=myapp -- sh

# Copy on same node
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --same-node --image=busybox
```

### Debug Node

```bash
# Interactive node debugging (host namespaces, filesystem at /host)
kubectl --context=my-context debug node/mynode -it --image=busybox

# With sysadmin profile for full capabilities
kubectl --context=my-context debug node/mynode -it --image=ubuntu --profile=sysadmin
```

## Debug Profiles

| Profile | Use Case | Capabilities |
|---------|----------|--------------|
| `legacy` | Default, unrestricted | Full access (backwards compatible) |
| `general` | General purpose | Moderate restrictions |
| `baseline` | Minimal restrictions | Pod security baseline |
| `netadmin` | Network troubleshooting | NET_ADMIN capability |
| `restricted` | High security environments | Strictest restrictions |
| `sysadmin` | System administration | SYS_PTRACE, SYS_ADMIN |

```bash
# Network debugging (tcpdump, netstat, ss)
kubectl --context=my-context debug mypod -it --image=nicolaka/netshoot --profile=netadmin

# System debugging (strace, perf)
kubectl --context=my-context debug mypod -it --image=ubuntu --profile=sysadmin
```

## Common Debug Images

| Image | Size | Use Case |
|-------|------|----------|
| `busybox` | ~1MB | Basic shell, common utilities |
| `alpine` | ~5MB | Shell with apk package manager |
| `ubuntu` | ~77MB | Full Linux with apt |
| `nicolaka/netshoot` | ~350MB | Network debugging (tcpdump, dig, curl, netstat) |
| `gcr.io/k8s-debug/debug` | Varies | Official Kubernetes debug image |

## Debugging Patterns

### Network Connectivity Issues

```bash
# Add netshoot container for network debugging
kubectl --context=my-context debug mypod -it \
  --image=nicolaka/netshoot \
  --profile=netadmin

# Inside container:
# - tcpdump -i any port 80
# - dig kubernetes.default
# - curl -v http://service:port
# - ss -tlnp
# - netstat -an
```

### Application Crashes

```bash
# Copy pod with different entrypoint to inspect
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --container=app \
  -- sh

# Inside: check filesystem, env vars, config files
```

### Process Inspection

```bash
# Target container's process namespace
kubectl --context=my-context debug mypod -it \
  --image=busybox \
  --target=mycontainer

# Inside: ps aux, /proc inspection
```

### Node-Level Issues

```bash
# Debug node with host access
kubectl --context=my-context debug node/worker-1 -it \
  --image=ubuntu \
  --profile=sysadmin

# Inside:
# - Host filesystem at /host
# - chroot /host for full access
# - journalctl, systemctl, dmesg
```

### Non-Destructive Debugging

```bash
# Create copy, keeping original running
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --same-node \
  --share-processes \
  --image=busybox

# Original pod continues serving traffic
# Debug copy shares storage if on same node
```

## Key Options

| Option | Description |
|--------|-------------|
| `-it` | Interactive TTY (required for shell access) |
| `--image` | Debug container image |
| `--container` | Name for the debug container |
| `--target` | Share process namespace with this container |
| `--copy-to` | Create a copy instead of ephemeral container |
| `--same-node` | Schedule copy on same node (with `--copy-to`) |
| `--set-image` | Change container images in copy |
| `--profile` | Security profile (legacy, netadmin, sysadmin, etc.) |
| `--share-processes` | Enable process namespace sharing (default: true with --copy-to) |
| `--replace` | Delete original pod when creating copy |

## Best Practices

1. **Use appropriate profiles** - Match capabilities to debugging needs
2. **Prefer ephemeral containers** - Less disruptive than pod copies
3. **Use `--copy-to` for invasive debugging** - Preserve original pod
4. **Clean up debug pods** - Delete copies after debugging
5. **Use `--same-node`** - For accessing shared storage/network conditions

## Cleanup

```bash
# List debug pod copies
kubectl --context=my-context get pods | grep -E "debug|copy"

# Delete debug pods
kubectl --context=my-context delete pod mypod-debug
```

## Requirements

- Kubernetes 1.23+ for ephemeral containers (stable)
- Kubernetes 1.25+ for debug profiles
- RBAC permissions for pods/ephemeralcontainers

For detailed option reference, examples, and troubleshooting patterns, see REFERENCE.md.