observability-patterns

What this skill does

# Observability Patterns Skill

This skill provides comprehensive templates and configurations for implementing observability in Google ADK agents. Includes logging, tracing, BigQuery analytics, Cloud Trace integration, and third-party observability platforms.

## Overview

Google ADK supports multiple observability approaches for monitoring, debugging, and analyzing agent behavior:

1. **Cloud Trace** - Google Cloud native tracing with OpenTelemetry
2. **BigQuery Agent Analytics** - Comprehensive event logging and analysis
3. **AgentOps** - Session replays and unified tracing analytics
4. **Phoenix (Arize)** - Open-source observability with self-hosted control
5. **Weave (W&B)** - Weights & Biases platform for tracking and visualization

This skill covers production-ready observability implementations with security and scalability.

## Available Scripts

### 1. Setup Cloud Trace

**Script**: `scripts/setup-cloud-trace.sh <project-id>`

**Purpose**: Configures Cloud Trace integration for ADK agents

**Parameters**:
- `project-id` - Google Cloud project ID (required)

**Usage**:
```bash
# Setup Cloud Trace for local development
./scripts/setup-cloud-trace.sh my-project-id

# Setup with ADK CLI deployment
adk deploy agent_engine --project=my-project-id --trace_to_cloud ./agent
```

**Environment Variables**:
- `GOOGLE_CLOUD_PROJECT` - Project ID for Cloud Trace
- `GOOGLE_APPLICATION_CREDENTIALS` - Path to service account key

**Output**: Cloud Trace enabled, traces visible in console.cloud.google.com

### 2. Setup BigQuery Agent Analytics

**Script**: `scripts/setup-bigquery-analytics.sh <project-id> <dataset-id> [bucket-name]`

**Purpose**: Configures BigQuery Agent Analytics plugin for comprehensive event logging

**Parameters**:
- `project-id` - Google Cloud project ID (required)
- `dataset-id` - BigQuery dataset name (required)
- `bucket-name` - GCS bucket for multimodal content (optional)

**Usage**:
```bash
# Setup basic BigQuery analytics
./scripts/setup-bigquery-analytics.sh my-project agent-analytics

# Setup with GCS for multimodal content
./scripts/setup-bigquery-analytics.sh my-project agent-analytics my-content-bucket

# Create dataset and table
bq mk --dataset my-project:agent-analytics
bq mk --table agent-analytics.agent_events_v2 templates/bigquery-schema.json
```

**IAM Requirements**:
- `roles/bigquery.jobUser` - Required for BigQuery operations
- `roles/bigquery.dataEditor` - Required for writing data
- `roles/storage.objectCreator` - Required if using GCS offloading

**Output**: BigQuery table created, events streaming to dataset

### 3. Setup AgentOps

**Script**: `scripts/setup-agentops.sh`

**Purpose**: Configures AgentOps integration for session replays and metrics

**Usage**:
```bash
# Install AgentOps
pip install -U agentops

# Setup with API key
AGENTOPS_API_KEY=your_api_key_here ./scripts/setup-agentops.sh

# Verify setup
python -c "import agentops; agentops.init(); print('AgentOps ready')"
```

**Environment Variables**:
- `AGENTOPS_API_KEY` - AgentOps API key from app.agentops.ai/settings/projects

**Output**: AgentOps initialized, sessions visible in dashboard

### 4. Setup Phoenix

**Script**: `scripts/setup-phoenix.sh`

**Purpose**: Configures Phoenix (Arize) integration for open-source observability

**Usage**:
```bash
# Install Phoenix packages
pip install openinference-instrumentation-google-adk arize-phoenix-otel

# Setup Phoenix with API key
PHOENIX_API_KEY=your_key_here \
PHOENIX_COLLECTOR_ENDPOINT=https://app.phoenix.arize.com/s/your-space \
./scripts/setup-phoenix.sh

# Verify Phoenix connection
python scripts/verify-phoenix.py
```

**Environment Variables**:
- `PHOENIX_API_KEY` - Phoenix API key from phoenix.arize.com
- `PHOENIX_COLLECTOR_ENDPOINT` - Phoenix collector endpoint URL

**Output**: Phoenix tracer initialized, traces visible in Phoenix dashboard

### 5. Setup Weave

**Script**: `scripts/setup-weave.sh <entity> <project>`

**Purpose**: Configures Weave (W&B) integration for observability

**Parameters**:
- `entity` - W&B entity name (visible in Teams sidebar)
- `project` - W&B project name

**Usage**:
```bash
# Install Weave dependencies
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# Setup Weave with API key
WANDB_API_KEY=your_wandb_key_here ./scripts/setup-weave.sh my-team my-project

# Verify Weave connection
python scripts/verify-weave.py
```

**Environment Variables**:
- `WANDB_API_KEY` - W&B API key from wandb.ai/authorize

**Output**: Weave tracer initialized, traces visible in Weave dashboard

### 6. Validate Observability Setup

**Script**: `scripts/validate-observability.sh`

**Purpose**: Validates observability configuration and connectivity

**Checks**:
- Cloud Trace connectivity
- BigQuery dataset and table existence
- AgentOps initialization
- Phoenix endpoint reachability
- Weave endpoint reachability
- IAM permissions
- Environment variables set

**Usage**:
```bash
# Validate all observability configurations
./scripts/validate-observability.sh

# Validate specific tool
./scripts/validate-observability.sh --tool=bigquery
./scripts/validate-observability.sh --tool=cloud-trace
./scripts/validate-observability.sh --tool=agentops
```

**Exit Codes**:
- `0` - All checks passed
- `1` - Configuration missing
- `2` - Connectivity failed
- `3` - Permission issues

## Available Templates

### 1. Cloud Trace Configuration

**Template**: `templates/cloud-trace-config.py`

**Purpose**: Cloud Trace integration for ADK agents

**Features**:
- OpenTelemetry configuration
- Automatic span creation for agent runs
- LLM and tool call tracing
- Error and latency tracking

**Usage**:
```python
# Enable Cloud Trace via ADK CLI
adk deploy agent_engine --project=$GOOGLE_CLOUD_PROJECT --trace_to_cloud ./agent

# Or via Python SDK
from google.adk.app import AdkApp

app = AdkApp(
    agent=my_agent,
    enable_tracing=True
)
```

**Span Labels**:
- `invocation` - Top-level agent invocation
- `agent_run` - Individual agent execution
- `call_llm` - LLM API calls
- `execute_tool` - Tool executions

### 2. BigQuery Analytics Configuration

**Template**: `templates/bigquery-analytics-config.py`

**Purpose**: Complete BigQuery Agent Analytics plugin configuration

**Features**:
- Asynchronous event logging
- Multimodal content with GCS offloading
- OpenTelemetry-style tracing (trace_id, span_id)
- Event filtering and batching
- Custom content formatting

**Usage**:
```python
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin, BigQueryLoggerConfig
)

bq_config = BigQueryLoggerConfig(
    enabled=True,
    gcs_bucket_name="your-bucket-name",
    max_content_length=500 * 1024,  # 500KB inline limit
    batch_size=1,  # Low latency
    event_allowlist=["LLM_RESPONSE", "TOOL_COMPLETED"]
)

plugin = BigQueryAgentAnalyticsPlugin(
    project_id="your-project-id",
    dataset_id="your-dataset-id",
    config=bq_config
)

app = App(root_agent=agent, plugins=[plugin])
```

**Configuration Options**:
- `enabled` - Toggle logging on/off
- `gcs_bucket_name` - GCS bucket for large content
- `max_content_length` - Inline text limit (default 500KB)
- `batch_size` - Events per write (default 1)
- `event_allowlist` - Whitelist specific event types
- `event_denylist` - Blacklist specific event types
- `content_formatter` - Custom formatting function

### 3. BigQuery Schema

**Template**: `templates/bigquery-schema.json`

**Purpose**: BigQuery table schema for agent_events_v2

**Schema Fields**:
- `timestamp` - Event recording time
- `event_type` - Event category (LLM_REQUEST, TOOL_STARTING, etc.)
- `content` - Event-specific JSON payload
- `content_parts` - Structured multimodal data
- `trace_id` - OpenTelemetry trace ID
- `span_id` - OpenTelemetry span ID
- `agent` - Agent name
- `user_id` - User identifier

**Partitioning**: By DATE(timestamp) for cost optimization

**Clustering**: By event_type, agent, user_id for query performance

### 4. AgentOps Configuration