shell-tools

What this skill does

# Shell Tools Skill

> Master jq, xargs, GNU parallel, and advanced pipelines

## Learning Objectives

After completing this skill, you will be able to:
- [ ] Process JSON with jq
- [ ] Use xargs for argument handling
- [ ] Parallelize tasks with GNU parallel
- [ ] Build efficient data pipelines
- [ ] Use utility commands effectively

## Prerequisites

- Strong Bash fundamentals
- Text processing basics
- Understanding of pipes

## Core Concepts

### 1. jq Essentials
```bash
# Basic queries
jq '.' file.json              # Pretty print
jq '.key' file.json           # Get key
jq '.array[0]' file.json      # First element
jq '.nested.key' file.json    # Nested

# Filtering
jq '.[] | select(.active)'    # Filter
jq '.[] | select(.count > 10)'

# Transform
jq '.[] | {id, name}'         # Select fields
jq 'map(.price * .qty)'       # Calculate
jq -r '.[] | @csv'            # To CSV

# From variables
jq -n --arg x "$VAR" '{value: $x}'
```

### 2. Xargs
```bash
# Basic usage
echo "a b c" | xargs echo

# Safe with spaces
find . -print0 | xargs -0 rm

# Limit arguments
cat list | xargs -n 1 process
cat list | xargs -n 10 process

# Parallel
cat list | xargs -P 4 -n 1 process

# Placeholder
cat urls | xargs -I {} curl {}
```

### 3. GNU Parallel
```bash
# Basic
parallel echo ::: a b c

# From file
parallel process :::: list.txt

# With options
parallel -j 4 process ::: *.txt
parallel --progress process ::: *.txt

# Complex
parallel -j 4 --delay 0.5 \
    'curl -s {} | jq .name' :::: urls.txt
```

### 4. Pipeline Utilities
```bash
# Sort and unique
sort file.txt
sort -n file.txt          # Numeric
sort -u file.txt          # Unique
sort file | uniq -c       # Count

# Cut and paste
cut -d',' -f1,3 file.csv
paste file1.txt file2.txt

# Transform
tr 'a-z' 'A-Z' < file
tr -d '\r' < dos.txt > unix.txt
```

## Common Patterns

### API Data Pipeline
```bash
curl -s 'https://api.example.com/users' |
    jq -r '.[] | select(.active) | [.id, .email] | @csv' |
    sort -t',' -k2 |
    head -20
```

### Parallel Processing
```bash
# Compress all logs in parallel
find . -name "*.log" |
    parallel -j 4 gzip

# Batch API calls with rate limit
cat ids.txt |
    parallel -j 5 --delay 0.2 \
        'curl -s "https://api.example.com/item/{}"'
```

### Data Transformation
```bash
# JSON to formatted output
cat data.json |
    jq -r '.items[] | "\(.id)\t\(.name)\t\(.price)"' |
    column -t
```

## Anti-Patterns

| Don't | Do | Why |
|-------|-----|-----|
| Parse JSON with grep | Use jq | Proper parsing |
| Sequential when parallel | Use parallel | Speed |
| `cat \| xargs` | `xargs < file` | Efficiency |

## Practice Exercises

1. **JSON Processor**: Transform API response
2. **Batch Processor**: Parallel file processing
3. **Log Analyzer**: Complex log pipeline
4. **Data Migrator**: Transform and load data

## Troubleshooting

### Common Errors

| Error | Cause | Fix |
|-------|-------|-----|
| `jq: error` | Invalid JSON | Validate with `jq .` |
| `xargs: arg too long` | Too many args | Use `-n` |
| `parallel: not found` | Not installed | `apt install parallel` |

### Debug Techniques
```bash
# Validate JSON
jq '.' < input.json

# Debug pipeline
command1 | tee /dev/stderr | command2

# Test jq filter
echo '{"a":1}' | jq '.a'
```

## Performance Tips

```bash
# Faster sorting
LC_ALL=C sort file.txt

# Parallel for CPU-bound
parallel -j $(nproc) process ::: *.txt

# Stream large files
jq -c '.[]' large.json | while read -r line; do
    # process line
done
```

## Resources

- [jq Manual](https://stedolan.github.io/jq/manual/)
- [GNU Parallel Tutorial](https://www.gnu.org/software/parallel/parallel_tutorial.html)
- [jq Playground](https://jqplay.org/)