gcp-gke-monitoring-observability

What this skill does

# GKE Monitoring and Observability

## Purpose

Implement comprehensive observability for GKE applications. This skill covers logging, metrics collection, visualization, distributed tracing, and alerting strategies for Spring Boot microservices.

## When to Use

Use this skill when you need to:
- Set up Cloud Logging and Cloud Monitoring for GKE applications
- Instrument Spring Boot applications with Actuator metrics
- Configure Prometheus scraping for custom metrics
- Create dashboards to visualize application performance
- Set up alerts for error rates, resource usage, or SLOs
- Enable distributed tracing with Cloud Trace
- Debug production issues using logs and metrics

Trigger phrases: "set up monitoring", "GKE observability", "configure Prometheus", "create dashboard", "set up alerts", "enable tracing"

## Table of Contents

- [Purpose](#purpose)
- [When to Use](#when-to-use)
- [Quick Start](#quick-start)
- [Instructions](#instructions)
  - [Step 1: Enable Cloud Operations Integration](#step-1-enable-cloud-operations-integration)
  - [Step 2: Configure Spring Boot Actuator](#step-2-configure-spring-boot-actuator)
  - [Step 3: Annotate Pods for Prometheus Scraping](#step-3-annotate-pods-for-prometheus-scraping)
  - [Step 4: View Logs](#step-4-view-logs)
  - [Step 5: Create Monitoring Dashboard](#step-5-create-monitoring-dashboard)
  - [Step 6: Set Up Alerts](#step-6-set-up-alerts)
  - [Step 7: Enable Distributed Tracing](#step-7-enable-distributed-tracing)
- [Examples](#examples)
- [Requirements](#requirements)
- [See Also](#see-also)

## Quick Start

Enable observability in three steps:

```bash
# 1. Enable Cloud Monitoring and Logging on cluster
gcloud container clusters update CLUSTER_NAME \
  --region=europe-west2 \
  --logging=SYSTEM,WORKLOAD \
  --monitoring=SYSTEM,WORKLOAD \
  --enable-managed-prometheus

# 2. Deploy Prometheus scrape config for Spring Boot Actuator
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: supplier-charges-hub
  namespace: wtr-supplier-charges
spec:
  ports:
  - name: metrics
    port: 8080
    targetPort: 8080
EOF

# 3. View logs and metrics in Cloud Console
gcloud logging read "resource.type=k8s_container AND resource.labels.namespace_name=wtr-supplier-charges" --limit=50
```

## Instructions

### Step 1: Enable Cloud Operations Integration

Configure the cluster to collect logs and metrics:

```bash
gcloud container clusters update shared-gke-labs-01-euw2 \
  --region=europe-west2 \
  --logging=SYSTEM,WORKLOAD \
  --monitoring=SYSTEM,WORKLOAD \
  --enable-managed-prometheus \
  --enable-cloud-logging \
  --enable-cloud-monitoring
```

**Components:**
- **Cloud Logging**: Captures container stdout/stderr
- **Cloud Monitoring**: System metrics (CPU, memory, disk)
- **Managed Service for Prometheus**: Application metrics (requires annotation)

### Step 2: Configure Spring Boot Actuator

Enable metrics and health endpoints in Spring Boot:

```yaml
# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus,env,configprops
  endpoint:
    health:
      probes:
        enabled: true
      show-details: always
    metrics:
      enabled: true
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true
    tags:
      application: supplier-charges-hub
      environment: labs
  health:
    livenessState:
      enabled: true
    readinessState:
      enabled: true
logging:
  pattern:
    console: '{"timestamp":"%d{ISO8601}","level":"%p","logger":"%c{1}","message":"%m"}%n'
```

### Step 3: Annotate Pods for Prometheus Scraping

Mark pods for metrics collection:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: supplier-charges-hub
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      containers:
      - name: supplier-charges-hub-container
        ports:
        - name: metrics
          containerPort: 8080
          protocol: TCP
```

### Step 4: View Logs

Query container logs from Cloud Logging:

```bash
# View recent logs
gcloud logging read "resource.type=k8s_container AND resource.labels.namespace_name=wtr-supplier-charges" \
  --limit=50 \
  --format=json | jq '.[] | {timestamp: .timestamp, message: .textPayload}'

# View logs with severity filter
gcloud logging read "resource.type=k8s_container AND resource.labels.namespace_name=wtr-supplier-charges AND severity=ERROR" \
  --limit=20

# View logs from specific pod
gcloud logging read "resource.type=k8s_pod AND resource.labels.pod_name=supplier-charges-hub-xyz123 AND resource.labels.namespace_name=wtr-supplier-charges" \
  --limit=50
```

**Alternative: View via kubectl:**

```bash
# Stream logs
kubectl logs -f deployment/supplier-charges-hub -n wtr-supplier-charges

# View logs from specific container (for multi-container pods)
kubectl logs deployment/supplier-charges-hub -c supplier-charges-hub-container -n wtr-supplier-charges

# View previous logs (after pod restart)
kubectl logs deployment/supplier-charges-hub -n wtr-supplier-charges --previous
```

### Step 5: Create Monitoring Dashboard

Visualize key metrics:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: supplier-charges-hub-metrics
  namespace: wtr-supplier-charges
spec:
  groups:
  - name: application-metrics
    interval: 30s
    rules:
    - alert: HighErrorRate
      expr: rate(http_server_requests_seconds_count{status=~"5.."}[5m]) > 0.05
      for: 5m
      annotations:
        summary: "High error rate detected"
    - alert: HighMemoryUsage
      expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85
      for: 10m
      annotations:
        summary: "Pod memory usage > 85%"
```

### Step 6: Set Up Alerts

Create alert policies for critical metrics:

```bash
# Create alert for high error rate
cat > alert-policy.yaml <<EOF
displayName: "Supplier Charges Hub - High Error Rate"
conditions:
  - displayName: "Error rate > 5%"
    conditionThreshold:
      filter: |
        resource.type="k8s_container"
        resource.namespace_name="wtr-supplier-charges"
        metric.type="logging.googleapis.com/user_defined_metric"
        metric.labels.severity="ERROR"
      comparison: COMPARISON_GT
      thresholdValue: 5
      duration: 300s
notificationChannels:
  - projects/ecp-wtr-supplier-charges-labs/notificationChannels/12345
EOF

# Deploy via gcloud (requires proper setup)
gcloud alpha monitoring policies create --policy-from-file=alert-policy.yaml
```

### Step 7: Enable Distributed Tracing

Add Spring Cloud Sleuth for request tracing:

```gradle
// build.gradle.kts
dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth")
    implementation("org.springframework.cloud:spring-cloud-sleuth-zipkin")
}
```

**Configuration:**

```yaml
# application.yml
spring:
  sleuth:
    sampler:
      probability: 0.1  # Sample 10% of requests
  zipkin:
    baseUrl: https://cloudtrace.googleapis.com  # Cloud Trace endpoint
```

**View traces:**

```bash
gcloud traces list --limit=10
gcloud traces describe TRACE_ID
```

## Examples

See [examples/examples.md](examples/examples.md) for comprehensive examples including:
- Complete observability setup script
- Log analysis and error tracking
- Health check endpoint testing
- Custom metrics dashboard creation

## Requirements

- GKE cluster with Cloud Logging and Cloud Monitoring enabled
- Spring Boot application with Actuator dependency
- `kubectl` access to the cluster
- `gcloud` CLI configured
- Service account with monitoring permissions: `roles/monitoring.metricWriter`

## See Also

- [gcp-gke-deployment-strategies](../gcp-gke-deployment-strategies/SKILL.md) - Understand health checks
- [gcp-gke-troubleshooting](../gcp-gke-troubleshooting/SKILL.md) - Use logs to debug issues
- [gcp-gke-cost-optimization](../gcp-gke-cost