shell-scripting

What this skill does

# Shell Scripting

Write reliable, maintainable bash scripts. Covers argument parsing, error handling, portability, temp files, parallel execution, process management, and self-documenting scripts.

## When to Use

- Writing scripts that others (or future you) will run
- Automating multi-step workflows
- Parsing command-line arguments with flags and options
- Handling errors and cleanup properly
- Running tasks in parallel
- Making scripts portable across Linux and macOS
- Wrapping complex commands with a simpler interface

## Script Template

```bash
#!/usr/bin/env bash
set -euo pipefail

# Description: What this script does (one line)
# Usage: script.sh [options] <required-arg>

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"

# Defaults
VERBOSE=false
OUTPUT_DIR="./output"

usage() {
    cat <<EOF
Usage: $SCRIPT_NAME [options] <input-file>

Description:
  Process the input file and generate output.

Options:
  -o, --output DIR    Output directory (default: $OUTPUT_DIR)
  -v, --verbose       Enable verbose output
  -h, --help          Show this help message

Examples:
  $SCRIPT_NAME data.csv
  $SCRIPT_NAME -v -o /tmp/results data.csv
EOF
}

log() { echo "[$(date '+%H:%M:%S')] $*" >&2; }
debug() { $VERBOSE && log "DEBUG: $*" || true; }
die() { log "ERROR: $*"; exit 1; }

# Parse arguments
while [[ $# -gt 0 ]]; do
    case "$1" in
        -o|--output) OUTPUT_DIR="$2"; shift 2 ;;
        -v|--verbose) VERBOSE=true; shift ;;
        -h|--help) usage; exit 0 ;;
        --) shift; break ;;
        -*) die "Unknown option: $1" ;;
        *) break ;;
    esac
done

INPUT_FILE="${1:?$(usage >&2; echo "Error: input file required")}"
[[ -f "$INPUT_FILE" ]] || die "File not found: $INPUT_FILE"

# Main logic
main() {
    debug "Input: $INPUT_FILE"
    debug "Output: $OUTPUT_DIR"
    mkdir -p "$OUTPUT_DIR"

    log "Processing $INPUT_FILE..."
    # ... do work ...
    log "Done. Output in $OUTPUT_DIR"
}

main "$@"
```

## Error Handling

### set flags

```bash
set -e          # Exit on any command failure
set -u          # Error on undefined variables
set -o pipefail # Pipe fails if any command in the pipe fails
set -x          # Debug: print each command before executing (noisy)

# Combined (use this in every script)
set -euo pipefail

# Temporarily disable for commands that are allowed to fail
set +e
some_command_that_might_fail
exit_code=$?
set -e
```

### Trap for cleanup

```bash
# Cleanup on exit (any exit: success, failure, or signal)
TMPDIR=""
cleanup() {
    [[ -n "$TMPDIR" ]] && rm -rf "$TMPDIR"
}
trap cleanup EXIT

TMPDIR=$(mktemp -d)
# Use $TMPDIR freely — it's cleaned up automatically

# Trap specific signals
trap 'echo "Interrupted"; exit 130' INT    # Ctrl+C
trap 'echo "Terminated"; exit 143' TERM    # kill
```

### Error handling patterns

```bash
# Check command exists before using it
command -v jq >/dev/null 2>&1 || die "jq is required but not installed"

# Provide default values
NAME="${NAME:-default_value}"

# Required variable (fail if unset)
: "${API_KEY:?Error: API_KEY environment variable is required}"

# Retry a command
retry() {
    local max_attempts=$1
    shift
    local attempt=1
    while [[ $attempt -le $max_attempts ]]; do
        "$@" && return 0
        log "Attempt $attempt/$max_attempts failed. Retrying..."
        ((attempt++))
        sleep $((attempt * 2))
    done
    die "Command failed after $max_attempts attempts: $*"
}

retry 3 curl -sf https://api.example.com/health
```

## Argument Parsing

### Simple: positional + flags

```bash
# Manual parsing (no dependencies)
FORCE=false
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case "$1" in
        -f|--force) FORCE=true; shift ;;
        -n|--dry-run) DRY_RUN=true; shift ;;
        -o|--output)
            [[ -n "${2:-}" ]] || die "--output requires a value"
            OUTPUT="$2"; shift 2 ;;
        --output=*)
            OUTPUT="${1#*=}"; shift ;;
        -h|--help) usage; exit 0 ;;
        --) shift; break ;;  # End of options
        -*) die "Unknown option: $1" ;;
        *) break ;;  # Start of positional args
    esac
done

# Remaining args are positional
FILES=("$@")
[[ ${#FILES[@]} -gt 0 ]] || die "At least one file is required"
```

### getopts (POSIX, short options only)

```bash
while getopts ":o:vhf" opt; do
    case "$opt" in
        o) OUTPUT="$OPTARG" ;;
        v) VERBOSE=true ;;
        f) FORCE=true ;;
        h) usage; exit 0 ;;
        :) die "Option -$OPTARG requires an argument" ;;
        ?) die "Unknown option: -$OPTARG" ;;
    esac
done
shift $((OPTIND - 1))
```

## Temp Files and Directories

```bash
# Create temp file (automatically unique)
TMPFILE=$(mktemp)
echo "data" > "$TMPFILE"

# Create temp directory
TMPDIR=$(mktemp -d)

# Create temp with custom prefix/suffix
TMPFILE=$(mktemp /tmp/myapp.XXXXXX)
TMPFILE=$(mktemp --suffix=.json)  # GNU only

# Always clean up with trap
trap 'rm -f "$TMPFILE"' EXIT

# Portable pattern (works on macOS and Linux)
TMPDIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'myapp')
trap 'rm -rf "$TMPDIR"' EXIT
```

## Parallel Execution

### xargs -P

```bash
# Run 4 commands in parallel
cat urls.txt | xargs -P 4 -I {} curl -sO {}

# Process files in parallel (4 at a time)
find . -name "*.csv" | xargs -P 4 -I {} ./process.sh {}

# Parallel with progress indicator
find . -name "*.jpg" | xargs -P 8 -I {} sh -c 'convert {} -resize 800x600 resized/{} && echo "Done: {}"'
```

### Background jobs + wait

```bash
# Run tasks in background, wait for all
pids=()
for file in data/*.csv; do
    process_file "$file" &
    pids+=($!)
done

# Wait for all and check results
failed=0
for pid in "${pids[@]}"; do
    wait "$pid" || ((failed++))
done
[[ $failed -eq 0 ]] || die "$failed jobs failed"
```

### GNU Parallel (if available)

```bash
# Process files with 8 parallel jobs
parallel -j 8 ./process.sh {} ::: data/*.csv

# With progress bar
parallel --bar -j 4 convert {} -resize 800x600 resized/{/} ::: *.jpg

# Pipe input lines
cat urls.txt | parallel -j 10 curl -sO {}
```

## Process Management

### Background processes

```bash
# Start in background
long_running_command &
BG_PID=$!

# Check if still running
kill -0 $BG_PID 2>/dev/null && echo "Running" || echo "Stopped"

# Wait for it
wait $BG_PID
echo "Exit code: $?"

# Kill on script exit
trap 'kill $BG_PID 2>/dev/null' EXIT
```

### Process supervision

```bash
# Run a command, restart if it dies
run_with_restart() {
    local cmd=("$@")
    while true; do
        "${cmd[@]}" &
        local pid=$!
        log "Started PID $pid"
        wait $pid
        local exit_code=$?
        log "Process exited with code $exit_code. Restarting in 5s..."
        sleep 5
    done
}

run_with_restart ./my-server --port 8080
```

### Timeout

```bash
# Kill command after 30 seconds
timeout 30 long_running_command

# With custom signal (SIGKILL after SIGTERM fails)
timeout --signal=TERM --kill-after=10 30 long_running_command

# Portable (no timeout command)
( sleep 30; kill $$ 2>/dev/null ) &
TIMER_PID=$!
long_running_command
kill $TIMER_PID 2>/dev/null
```

## Portability (Linux vs macOS)

### Common differences

```bash
# sed: macOS requires -i '' (empty backup extension)
# Linux:
sed -i 's/old/new/g' file.txt
# macOS:
sed -i '' 's/old/new/g' file.txt
# Portable:
sed -i.bak 's/old/new/g' file.txt && rm file.txt.bak

# date: different flags
# GNU (Linux):
date -d '2026-02-03' '+%s'
# BSD (macOS):
date -j -f '%Y-%m-%d' '2026-02-03' '+%s'

# readlink -f: doesn't exist on macOS
# Portable alternative:
real_path() { cd "$(dirname "$1")" && echo "$(pwd)/$(basename "$1")"; }

# stat: different syntax
# GNU: stat -c '%s' file
# BSD: stat -f '%z' file

# grep -P: not available on macOS by default
# Use grep -E instead, or install GNU grep
```

### POSIX-safe patterns

```bash
# Use printf instead of echo -e (echo behavior varies)
printf "Line 1\nLine 2\n"

# Use $() instead of backticks
result=$(command)   # Good
result=`command`    # Bad (de