jq-json-processing

What this skill does

# jq JSON Processing

Expert knowledge for processing, querying, and transforming JSON data using jq, the lightweight and flexible command-line JSON processor.

## When to Use This Skill

| Use this skill when... | Use yq-yaml-processing instead when... |
|---|---|
| Extracting or filtering fields in JSON input | Working with YAML files (Kubernetes, GH Actions, Helm) |
| Parsing `gh`, `curl`, or `kubectl -o json` responses | Editing YAML in place while preserving comments |
| CI pipelines on minimal images (jq is universally installed) | Converting YAML → JSON before processing |

| Use this skill when... | Use nushell-data-processing instead when... |
|---|---|
| One-off pipeline transforms that fit in a single expression | Multi-step transforms that span JSON, YAML, CSV, and TOML |
| Stdin-piped JSON from another command | Aggregations, group-by, or visual table exploration |

## Core Expertise

**JSON Operations**
- Query and filter JSON with path expressions
- Transform JSON structure and shape
- Combine, merge, and split JSON data
- Validate JSON syntax and structure

**Data Extraction**
- Extract specific fields from JSON objects
- Filter arrays based on conditions
- Navigate nested JSON structures
- Handle null values and missing keys

## Essential Commands

### Basic Querying

```bash
# Pretty-print JSON
jq '.' file.json

# Extract specific field
jq '.fieldName' file.json

# Extract nested field
jq '.user.email' file.json

# Extract array element
jq '.[0]' file.json
jq '.items[2]' file.json
```

### Array Operations

```bash
# Get array length
jq '.items | length' file.json

# Map over array
jq '.items | map(.name)' file.json

# Filter array
jq '.items[] | select(.active == true)' file.json
jq '.users[] | select(.age > 18)' file.json

# Sort array
jq '.items | sort_by(.name)' file.json
jq '.items | sort_by(.date) | reverse' file.json

# Get first/last elements
jq '.items | first' file.json
jq '.items | last' file.json

# Unique values
jq '.tags | unique' file.json
```

### Object Operations

```bash
# Get all keys
jq 'keys' file.json
jq '.config | keys' file.json

# Get all values
jq '.[] | values' file.json

# Select specific fields
jq '{name, email}' file.json
jq '{name: .fullName, id: .userId}' file.json

# Add field
jq '. + {newField: "value"}' file.json

# Delete field
jq 'del(.password)' file.json

# Merge objects
jq '. * {updated: true}' file.json
```

### Filtering and Conditions

```bash
# Select with conditions
jq 'select(.status == "active")' file.json
jq '.[] | select(.price < 100)' file.json

# Multiple conditions (AND)
jq '.[] | select(.active and .verified)' file.json
jq '.[] | select(.age > 18 and .country == "US")' file.json

# Multiple conditions (OR)
jq '.[] | select(.type == "admin" or .type == "moderator")' file.json

# Exists / has field
jq '.[] | select(has("email"))' file.json
jq 'select(.optional != null)' file.json

# Not condition
jq '.[] | select(.status != "deleted")' file.json
```

### String Operations

```bash
# String interpolation
jq '"Hello, \(.name)"' file.json

# Convert to string
jq '.id | tostring' file.json

# String contains
jq '.[] | select(.email | contains("@gmail.com"))' file.json

# String starts/ends with
jq '.[] | select(.name | startswith("A"))' file.json
jq '.[] | select(.file | endswith(".json"))' file.json

# Split string
jq '.path | split("/")' file.json

# Join array to string
jq '.tags | join(", ")' file.json

# Lowercase/uppercase
jq '.name | ascii_downcase' file.json
jq '.name | ascii_upcase' file.json
```

### Pipes and Composition

```bash
# Chain operations
jq '.items | map(.name) | sort | unique' file.json

# Multiple filters
jq '.users[] | select(.active) | select(.age > 18) | .email' file.json

# Group by
jq 'group_by(.category)' file.json
jq 'group_by(.status) | map({status: .[0].status, count: length})' file.json
```

### Output Formatting

```bash
# Compact output (no pretty-print)
jq -c '.' file.json

# Raw output (no quotes for strings)
jq -r '.message' file.json
jq -r '.items[] | .name' file.json

# Output as tab-separated values
jq -r '.[] | [.name, .age, .email] | @tsv' file.json

# Output as CSV
jq -r '.[] | [.name, .age, .email] | @csv' file.json

# Output as JSON array on one line
jq -c '[.items[]]' file.json
```

### Input/Output Options

```bash
# Read from stdin
cat file.json | jq '.items'
curl -s https://api.example.com/data | jq '.results'

# Multiple input files
jq -s '.' file1.json file2.json  # Slurp into array

# Write to file
jq '.filtered' input.json > output.json

# In-place edit (use sponge from moreutils)
jq '.updated = true' file.json | sponge file.json
```

### Advanced Patterns

```bash
# Recursive descent
jq '.. | .email? // empty' file.json

# Reduce (sum, accumulate)
jq '[.items[].price] | add' file.json
jq 'reduce .items[] as $item (0; . + $item.price)' file.json

# Variable assignment
jq '.items[] | . as $item | $item.name + " - " + ($item.price | tostring)' file.json

# Conditional (if-then-else)
jq '.items[] | if .price > 100 then "expensive" else "affordable" end' file.json
jq 'if .error then .error else .data end' file.json

# Try-catch for error handling
jq '.items[] | try .field catch "not found"' file.json

# Flatten nested arrays
jq '.items | flatten' file.json
jq '.items | flatten(1)' file.json  # Flatten one level
```

### Real-World Examples

```bash
# Extract all emails from nested structure
jq '.. | .email? // empty' users.json

# Get unique list of all tags across items
jq '[.items[].tags[]] | unique' data.json

# Count items by status
jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json

# Transform API response to simple list
jq '.results[] | {id, name: .full_name, active: .is_active}' response.json

# Filter GitHub workflow runs (recent failures)
gh run list --json status,conclusion,name,createdAt | \
  jq '.[] | select(.conclusion == "failure") | {name, createdAt}'

# Extract package.json dependencies with versions
jq '.dependencies | to_entries | map("\(.key)@\(.value)")' package.json

# Merge two JSON files
jq -s '.[0] * .[1]' base.json override.json

# Create summary from log data
jq 'group_by(.level) | map({level: .[0].level, count: length, samples: [.[].message][:3]})' logs.json
```

## Best Practices

**Query Construction**
- Start simple, build complexity incrementally
- Test filters on small datasets first
- Use `-c` flag for compact output in scripts
- Use `-r` flag for raw strings (no quotes)

**Performance**
- Use `select()` early in pipeline to reduce data
- Use targeted queries with specific paths for large files
- Stream large JSON with `--stream` flag
- Consider `jq -c` for faster processing

**Error Handling**
- Use `?` operator for optional access: `.field?`
- Use `// empty` to filter out nulls/errors
- Use `try-catch` for graceful error handling
- Check for field existence with `has("field")`

**Readability**
- Break complex queries into multiple steps
- Use variables with `as $var` for clarity
- Add comments in shell scripts
- Format multi-line jq programs for readability

## Common Patterns

### API Response Processing
```bash
# GitHub API: Get PR titles and authors
gh pr list --json title,author,number | \
  jq -r '.[] | "#\(.number) - \(.title) by @\(.author.login)"'

# REST API: Extract and flatten pagination
curl -s "https://api.example.com/items" | \
  jq '.data.items[] | {id, name, status}'
```

### Configuration Files
```bash
# Extract environment-specific config
jq '.environments.production' config.json

# Update configuration value
jq '.settings.timeout = 30' config.json > config.updated.json

# Merge base config with environment overrides
jq -s '.[0] * .[1]' base-config.json prod-config.json
```

### Log Analysis
```bash
# Count errors by type
jq 'select(.level == "error") | .type' logs.json | sort | uniq -c

# Extract error messages with timestamps
jq -r 'select(.level == "error") | "\(.timestamp) - \(.message)"' logs.json

# Group by hour and count
jq -r '.timestamp | spl