operating-kubernetes

What this skill does

# Kubernetes Operations

## Purpose

Operating Kubernetes clusters in production requires mastery of resource management, scheduling patterns, networking architecture, storage strategies, security hardening, and autoscaling. This skill provides operations-first frameworks for right-sizing workloads, implementing high-availability patterns, securing clusters with RBAC and Pod Security Standards, and systematically troubleshooting common failures.

Use this skill when deploying applications to Kubernetes, configuring cluster resources, implementing NetworkPolicies for zero-trust security, setting up autoscaling (HPA, VPA, KEDA), managing persistent storage, or diagnosing operational issues like CrashLoopBackOff or resource exhaustion.

## When to Use This Skill

**Common Triggers:**
- "Deploy my application to Kubernetes"
- "Configure resource requests and limits"
- "Set up autoscaling for my pods"
- "Implement NetworkPolicies for security"
- "My pod is stuck in Pending/CrashLoopBackOff"
- "Configure RBAC with least privilege"
- "Set up persistent storage for my database"
- "Spread pods across availability zones"

**Operations Covered:**
- Resource management (CPU/memory, QoS classes, quotas)
- Advanced scheduling (affinity, taints, topology spread)
- Networking (NetworkPolicies, Ingress, Gateway API)
- Storage operations (StorageClasses, PVCs, CSI)
- Security hardening (RBAC, Pod Security Standards, policies)
- Autoscaling (HPA, VPA, KEDA, cluster autoscaler)
- Troubleshooting (systematic debugging playbooks)

## Resource Management

### Quality of Service (QoS) Classes

Kubernetes assigns QoS classes based on resource requests and limits:

**Guaranteed (Highest Priority):**
- Requests equal limits for CPU and memory
- Never evicted unless exceeding limits
- Use for critical production services

```yaml
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"  # Same as request
    cpu: "500m"
```

**Burstable (Medium Priority):**
- Requests less than limits (or only requests set)
- Can burst above requests
- Evicted under node pressure
- Use for web servers, most applications

```yaml
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"  # 2x request
    cpu: "500m"
```

**BestEffort (Lowest Priority):**
- No requests or limits set
- First to be evicted under pressure
- Use only for development/testing

### Decision Framework: Which QoS Class?

| Workload Type | QoS Class | Configuration |
|---------------|-----------|---------------|
| Critical API/Database | Guaranteed | requests == limits |
| Web servers, services | Burstable | limits 1.5-2x requests |
| Batch jobs | Burstable | Low requests, high limits |
| Dev/test environments | BestEffort | No limits |

### Resource Quotas and LimitRanges

Enforce multi-tenancy with ResourceQuotas (namespace limits) and LimitRanges (per-container defaults):

```yaml
# ResourceQuota: Namespace-level limits
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"
```

For detailed resource management patterns including Vertical Pod Autoscaler (VPA), see `references/resource-management.md`.

## Advanced Scheduling

### Node Affinity

Control which nodes pods schedule on with required (hard) or preferred (soft) constraints:

```yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - g4dn.xlarge  # GPU instance
```

### Taints and Tolerations

Reserve nodes for specific workloads (inverse of affinity):

```bash
# Taint GPU nodes to prevent non-GPU workloads
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule
```

```yaml
# Pod tolerates GPU taint
tolerations:
- key: "workload"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"
```

### Topology Spread Constraints

Distribute pods evenly across failure domains (zones, nodes):

```yaml
topologySpreadConstraints:
- maxSkew: 1  # Max difference in pod count
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app
```

For advanced scheduling patterns including pod priority and preemption, see `references/scheduling-patterns.md`.

## Networking

### NetworkPolicies (Zero-Trust Security)

Implement default-deny security with NetworkPolicies:

```yaml
# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
```

```yaml
# Allow specific ingress (frontend → backend)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
```

### Ingress vs. Gateway API

**Ingress (Legacy):**
- Widely supported, mature ecosystem
- Limited expressiveness
- Use for existing applications

**Gateway API (Modern):**
- Role-oriented design (cluster ops vs. app devs)
- More expressive (HTTPRoute, TCPRoute, TLSRoute)
- Recommended for new applications (GA in Kubernetes 1.29+)

```yaml
# Gateway API example
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: backend
      port: 8080
```

For detailed networking patterns including service mesh integration, see `references/networking.md`.

## Storage

### StorageClasses (Define Performance Tiers)

StorageClasses define storage tiers for different workload needs:

```yaml
# AWS EBS SSD (high performance)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iopsPerGB: "50"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
```

### Storage Decision Matrix

| Workload | Performance | Access Mode | Storage Class |
|----------|-------------|-------------|---------------|
| Database | High | ReadWriteOnce | SSD (gp3/io2) |
| Shared files | Medium | ReadWriteMany | NFS/EFS |
| Logs (temp) | Low | ReadWriteOnce | Standard HDD |
| ML models | High | ReadOnlyMany | Object storage (S3) |

**Access Modes:**
- **ReadWriteOnce (RWO):** Single node read-write (most common)
- **ReadOnlyMany (ROX):** Multiple nodes read-only
- **ReadWriteMany (RWX):** Multiple nodes read-write (requires network storage)

For detailed storage operations including volume snapshots and CSI drivers, see `references/storage.md`.

## Security

### RBAC (Role-Based Access Control)

Implement least-privilege access with RBAC:

```yaml
# Role (namespace-scoped)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
# RoleBinding (assign role to user)
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
```

### Pod Security Standards

Enforce secure pod configurations at the namespace level:

```yaml
# Namespace with Restricted PSS (most secure)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restrict