observability-analyze-logs

What this skill does

# Analyze Logs

## Table of Contents

### Core Sections
- [What This Skill Does](#what-this-skill-does) - Intelligent OpenTelemetry log analysis with trace reconstruction
- [When to Use This Skill](#when-to-use-this-skill) - Trigger phrases and common scenarios
- [Quick Start](#quick-start) - Most common usage for quick health checks
- [Analysis Workflow](#analysis-workflow) - Complete step-by-step implementation guide
  - [Step 1: Determine User's Need](#step-1-determine-users-need) - Identify analysis type (health check, error investigation, trace debugging)
  - [Step 2: Choose Analysis Mode](#step-2-choose-analysis-mode) - 6 modes: Summary, Error Detail, Trace Analysis, File Filter, Fast Parsing, Real-Time
  - [Step 3: Execute Analysis](#step-3-execute-analysis) - Running commands with Bash tool
  - [Step 4: Interpret Results](#step-4-interpret-results) - Understanding summary tables, detailed output, AI analysis
  - [Step 5: Report Findings](#step-5-report-findings) - Communicating results to users
- [Command Reference](#command-reference) - All analyzer commands with examples
  - Basic Commands (summary, markdown, JSON)
  - Filtering Commands (--error-id, --trace, --file)
  - Performance Commands (--no-ai, --output, tail)
- [Understanding the Output](#understanding-the-output) - Log format, summary tables, trace execution
- [Best Practices](#best-practices) - 7 essential practices for effective log analysis
- [Common Scenarios](#common-scenarios) - Real-world examples with commands
- [Integration with CLAUDE.md](#integration-with-claudemd) - How this skill implements CLAUDE.md conventions

### Advanced Topics
- [Troubleshooting](#troubleshooting) - Common issues and fixes
- [Advanced Usage](#advanced-usage) - Custom scripts, programmatic access, budget tracking
- [Expected Outcomes](#expected-outcomes) - Success criteria and common results
- [Requirements](#requirements) - Tools, files, dependencies, verification

### Supporting Resources
- [Technical Reference](references/reference.md) - OpenTelemetry format, data models, parsing logic, AI configuration
- [Response Templates](templates/response-template.md) - 8 response templates for different contexts
- [Related Documentation](#related-documentation) - Tool implementation, usage examples, LangChain client
- [Quick Reference Card](#quick-reference-card) - One-line command cheatsheet

---

## Purpose

Intelligent log analysis for any project using OpenTelemetry trace reconstruction and AI-powered error diagnosis. Parses OTEL-formatted logs, reconstructs execution traces, extracts errors with call chain context, and provides root cause analysis.

## What This Skill Does

Intelligent log analysis for any project using OpenTelemetry trace reconstruction and AI-powered error diagnosis. Works with projects that generate OpenTelemetry-formatted logs in a configurable log directory.

**Core capabilities:**
- Parse OpenTelemetry-formatted logs with trace/span IDs
- Reconstruct complete execution traces
- Extract errors with full call chain context
- AI-powered root cause analysis
- Multiple output formats (summary, markdown, JSON)
- Advanced filtering (by error ID, trace ID, file)

## When to Use This Skill

Invoke this skill when users mention:
- "check the logs"
- "look at the logs"
- "analyze errors"
- "what's failing?"
- "debug this issue"
- "show me the traces"
- "investigate the error"
- "view log file"
- Any mention of project log files ({{LOG_DIR}}/{{LOG_FILE}}.log)

## Quick Start

**Most common usage (quick health check):**
```bash
python3 .claude/tools/utils/log_analyzer.py {{LOG_DIR}}/{{LOG_FILE}}.log
```

This gives you a summary table with error IDs and trace IDs, perfect for quick health checks. Replace `{{LOG_DIR}}` with your project's log directory (e.g., `logs`) and `{{LOG_FILE}}` with the log filename (e.g., `app.log`).

## Instructions

### Step 1: Determine User's Need

**Quick Health Check:**
- User asks: "Are there any errors?" or "What's happening in the logs?"
- Action: Run summary mode (default)

**Specific Error Investigation:**
- User mentions specific error or asks for details
- Action: Get error ID from summary, then use --error-id

**Trace-Based Debugging:**
- User asks "what led to this error?" or wants execution flow
- Action: Use --trace with trace ID

**File-Specific Analysis:**
- User mentions specific file or module
- Action: Use --file filter

**Real-Time Monitoring:**
- User wants to watch logs as they happen
- Action: Use tail -f

### Step 2: Choose Analysis Mode

**Mode 1: Summary (Default) - Start here 90% of the time**
```bash
python3 .claude/tools/utils/log_analyzer.py {{LOG_DIR}}/{{LOG_FILE}}.log
```

Output: Compact table with error IDs, trace IDs, file:line, function, and message preview.

**Use when:**
- Initial investigation
- Quick health check
- Getting error IDs for deeper analysis
- User asks "what errors do we have?"

**Mode 2: Error Detail - Deep dive into specific error**
```bash
python3 .claude/tools/utils/log_analyzer.py --error-id 1 --format markdown
```

Output: Full error details including complete message, call chain, stack trace, related errors.

**Use when:**
- User asks about specific error from summary
- Need full error message (summary truncates)
- Want to see complete stack trace
- Investigating single failure

**Mode 3: Trace Analysis - Understand execution flow**
```bash
python3 .claude/tools/utils/log_analyzer.py --trace TRACE_ID --format markdown
```

Output: All errors in that trace with full execution context.

**Use when:**
- Multiple related errors in same trace
- Need to understand execution sequence
- Debugging distributed operations
- User asks "what happened in this execution?"

**Mode 4: File Filter - Find all errors in specific file**
```bash
python3 .claude/tools/utils/log_analyzer.py --file database.py --format markdown
```

Output: All errors from that file with trace context.

**Use when:**
- User mentions specific file
- Investigating module-specific issues
- Finding patterns in one component

**Mode 5: Fast Parsing (No AI) - Quick and free**
```bash
python3 .claude/tools/utils/log_analyzer.py --no-ai
```

Output: Same as summary but skips AI analysis (faster, no cost).

**Use when:**
- Quick checks during development
- Want to avoid LLM costs
- Just need parsed errors without analysis
- Automated scripts or frequent polling

**Mode 6: Real-Time Monitoring**
```bash
tail -f {{LOG_DIR}}/{{LOG_FILE}}.log
```

Output: Live log stream (Ctrl+C to exit).

**Use when:**
- Watching logs during testing
- Monitoring server startup
- Debugging in real-time
- User runs operations and wants to see results

### Step 3: Execute Analysis

**Execute the appropriate command using the Bash tool:**

```bash
# Example for summary
python3 .claude/tools/utils/log_analyzer.py {{LOG_DIR}}/{{LOG_FILE}}.log
```

### Step 4: Interpret Results

**For Summary Output:**
1. Check "Total errors" count
2. Scan error table for patterns (same file, same trace)
3. Note error IDs for deeper investigation
4. Use provided Quick Commands to drill down

**For Detailed Output:**
1. Read Full Message (not truncated)
2. Review Call Chain (execution flow leading to error)
3. Check Related Errors (other failures in same trace)
4. Examine Stack Trace (if available)
5. Look for Recovery Attempts (logs after error)

**For AI Analysis (markdown with --no-ai not set):**
1. Read Root Causes section
2. Check Patterns (recurring issues)
3. Review Priority (what to fix first)
4. Follow Fixes (specific file:line changes)
5. Consider Systemic Issues (larger architectural problems)

### Step 5: Report Findings

**Always provide:**
1. Summary of error count and severity
2. Most critical issues (from AI analysis or your judgment)
3. Specific file:line references for user to investigate
4. Suggested next steps or commands to run

**Example response format:**
```
Found 5 errors in the logs:

Critical Issues:
1. Neo4j connection failure in database.py:123 (trace: abc123)
   - A