log-ops

What this skill does

# Log Operations

Practical patterns for analyzing log files -- especially JSONL format used in agent conversation logs, benchmark outputs, and structured application logs.

## Log Format Decision Tree

```
Unknown Log File
│
├─ Is it one JSON object per line?
│  ├─ Yes ──────────────────────── JSONL
│  │  ├─ Small file (<100MB)
│  │  │  └─ jq for extraction, jq -s for aggregation
│  │  ├─ Large file (100MB-1GB)
│  │  │  └─ rg prefilter then pipe to jq
│  │  └─ Huge file (>1GB)
│  │     └─ split + parallel jq, or jq --stream
│  │
│  └─ No
│     ├─ Is it one large JSON object/array?
│     │  └─ Yes ──────────────── Single JSON
│     │     └─ jq --stream for SAX-style, or jq directly if fits in memory
│     │
│     ├─ Does it have key=value pairs?
│     │  └─ Yes ──────────────── Structured (logfmt / key-value)
│     │     └─ rg for search, awk/sd for extraction, angle-grinder for aggregation
│     │
│     ├─ Does it follow syslog format? (timestamp hostname service[pid]: message)
│     │  └─ Yes ──────────────── Syslog
│     │     └─ rg for search, awk for column extraction, lnav for interactive
│     │
│     ├─ Is it space/tab delimited with consistent columns?
│     │  └─ Yes ──────────────── Column-based (access logs, CSV)
│     │     └─ awk for extraction, mlr for CSV, rg for pattern search
│     │
│     └─ Mixed or unstructured
│        └─ Plain text ─────────── Freeform
│           └─ rg for search, rg -A/-B for context, lnav for exploration
```

## Prerequisites

**Required** (must be installed):
- `rg` (ripgrep) - text search, prefiltering. Install: `cargo install ripgrep` / `choco install ripgrep`
- `jq` - JSON/JSONL extraction and transformation. Install: `brew install jq` / `choco install jq`

**Optional** (enhanced capabilities, gracefully degraded without):
- `lnav` - interactive log exploration with SQL queries. Install: `brew install lnav` / WSL: `apt install lnav`
- `agrind` (angle-grinder) - pipeline aggregation syntax. Install: `cargo install ag`
- `mlr` (Miller) - CSV/TSV log analysis. Install: `brew install miller` / `choco install miller`
- `GNU parallel` - parallel processing of split files. Install: `brew install parallel`

> All patterns in this skill work with just rg + jq. Optional tools add interactive exploration (lnav), pipeline aggregation (agrind), and tabular analysis (mlr).

## Tool Selection Matrix

| Tool | Best For | Speed | Required? |
|------|----------|-------|-----------|
| `rg` (ripgrep) | Raw pattern matching in any format | Fastest | Yes |
| `jq` | JSONL structured extraction and transformation | Fast | Yes |
| `jq -s` | JSONL aggregation (slurp all lines into array) | Medium (loads all into memory) | Yes (part of jq) |
| `lnav` | Interactive exploration, SQL over logs | Interactive | Optional |
| `agrind` (angle-grinder) | Pipeline aggregation and counting | Fast | Optional |
| `awk` | Column-based log formats, field extraction | Fast | Pre-installed |
| `mlr` (Miller) | CSV/TSV log analysis, statistics | Fast | Optional |
| `fd` + `rg` | Searching across many log directories | Fast | Pre-installed in dev-shell |
| `GNU parallel` | Splitting large files for parallel processing | N/A (orchestrator) | Optional |

### When to Use What

```
Need to...
│
├─ Find lines matching a pattern
│  └─ rg (always fastest for text search)
│
├─ Extract specific fields from JSONL
│  └─ jq -r '[.field1, .field2] | @tsv'
│
├─ Count/aggregate over JSONL
│  └─ jq -sc 'group_by(.field) | map({key: .[0].field, n: length})'
│
├─ Search JSONL by value then format results
│  └─ rg '"error"' file.jsonl | jq -r '.message'  (two-stage)
│
├─ Explore interactively with filtering/SQL
│  └─ lnav file.log
│
├─ Aggregate with pipeline syntax
│  └─ agrind '* | parse "* * *" as ts, level, msg | count by level'
│
├─ Extract columns from space-delimited logs
│  └─ awk '{print $1, $4, $7}' access.log
│
└─ Process CSV/TSV logs with headers
   └─ mlr --csv filter '$status >= 400' then stats1 -a count -f status
```

## JSONL Quick Reference

The most common format for structured logs. One JSON object per line, no trailing commas, no wrapping array.

### Stream Filtering (line by line, constant memory)

```bash
# Filter by field value
jq -c 'select(.level == "error")' app.jsonl

# Filter by nested field
jq -c 'select(.request.method == "POST")' app.jsonl

# Filter by multiple conditions
jq -c 'select(.level == "error" and .status >= 500)' app.jsonl

# Filter by array contains
jq -c 'select(.tags | index("critical"))' app.jsonl

# Filter by field existence
jq -c 'select(.stack_trace != null)' app.jsonl

# Negate a filter
jq -c 'select(.level != "debug")' app.jsonl
```

### Field Extraction

```bash
# Extract single field
jq -r '.message' app.jsonl

# Extract multiple fields as TSV
jq -r '[.timestamp, .level, .message] | @tsv' app.jsonl

# Extract with default for missing fields
jq -r '.error_code // "none"' app.jsonl

# Extract nested field safely
jq -r '.response.headers["content-type"] // "unknown"' app.jsonl
```

### Aggregation (requires slurp: loads entire file)

```bash
# Count by field value
jq -sc 'group_by(.level) | map({level: .[0].level, count: length})' app.jsonl

# Top-N most common values
jq -sc '[.[].error_type] | group_by(.) | map({type: .[0], count: length}) | sort_by(-.count) | .[:10]' app.jsonl

# Sum a numeric field
jq -sc 'map(.duration_ms) | add' app.jsonl

# Average
jq -sc 'map(.duration_ms) | add / length' app.jsonl

# Min and max
jq -sc 'map(.duration_ms) | {min: min, max: max}' app.jsonl
```

### Nested Extraction (agent logs, complex structures)

```bash
# Extract tool calls from conversation logs
jq -c '.content[]? | select(.type == "tool_use") | .name' conversation.jsonl

# De-escape nested JSON strings
jq -c '.content | fromjson' app.jsonl

# Flatten nested arrays
jq -c '[.events[]? | .action]' app.jsonl

# Extract from arrays of objects
jq -c '.results[]? | select(.passed == false) | {test: .name, error: .message}' results.jsonl
```

### Two-Stage Pipeline (rg for speed, jq for structure)

```bash
# Fast prefilter then structured extraction
rg '"error"' app.jsonl | jq -r '[.timestamp, .message] | @tsv'

# Search for specific value then aggregate
rg '"timeout"' app.jsonl | jq -sc 'length'

# Pattern match then extract
rg '"user_id":"u-123"' app.jsonl | jq -c '{ts: .timestamp, action: .action}'
```

### Time-Range Filtering

```bash
# Filter by timestamp range (ISO 8601 string comparison works)
jq -c 'select(.timestamp > "2026-03-08T10:00" and .timestamp < "2026-03-08T11:00")' app.jsonl

# Events in the last N minutes (using epoch seconds)
jq -c --arg cutoff "$(date -d '30 minutes ago' +%s)" 'select((.timestamp | sub("\\.[0-9]+Z$"; "Z") | fromdate) > ($cutoff | tonumber))' app.jsonl

# Extract hour for histogram
jq -r '.timestamp | split("T")[1] | split(":")[0]' app.jsonl | sort | uniq -c
```

### Cross-File Join

```bash
# Extract IDs from one file, search in another
jq -r '.request_id' errors.jsonl | while read id; do
  rg "\"$id\"" responses.jsonl | jq -c '{id: .request_id, status: .status}'
done

# Faster: build lookup, then join
jq -r '.request_id' errors.jsonl | sort -u > /tmp/error_ids.txt
rg -Ff /tmp/error_ids.txt responses.jsonl | jq -c '{id: .request_id, status: .status}'

# Join two JSONL files by key using jq --slurpfile
jq --slurpfile lookup <(jq -sc 'map({(.id): .}) | add' lookup.jsonl) \
  '. + ($lookup[0][.ref_id] // {})' main.jsonl
```

## Plain Text Log Patterns

### Pattern Search with Context

```bash
# Show 5 lines before and after each match
rg -B5 -A5 "OutOfMemoryError" app.log

# Show only matching files
rg -l "FATAL" /var/log/

# Count matches per file
rg -c "ERROR" /var/log/*.log | sort -t: -k2 -rn

# Multiline patterns (stack traces)
rg -U "Exception.*\n(\s+at .*\n)+" app.log
```

### Column Extraction with awk

```bash
# Apache/nginx access log: extract status codes
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Extract specific time range from syslog
awk '$0 >= "Mar  8 10:00" && $0