shell-scripting

What this skill does

# Bash Scripting Best Practices

Guidance for writing reliable, maintainable bash scripts following modern best practices. Emphasises simplicity, automated tooling, and defensive programming without over-engineering.

## When to Use Shell (and When Not To)

### Use Shell For:
- Small utilities and simple wrapper scripts (<100 lines)
- Orchestrating other programmes and tools
- Simple automation tasks
- Build/deployment scripts with straightforward logic
- Quick data transformation pipelines

### Do NOT Use Shell For:
- Complex business logic or data structures
- Performance-critical code
- Scripts requiring extensive error handling
- Anything over ~100 lines or with non-straightforward control flow
- When you need proper data structures beyond arrays

**Critical**: If your script grows too large (1000+ lines) or complex, consider offering to rewrite it in a proper language (Python, Go, etc.) before it becomes unmaintainable.

## Mandatory Foundations

Every bash script must have these elements:

### 1. Proper Shebang
```bash
#!/usr/bin/env bash
```
**Why**: Portable across systems where bash may not be at `/bin/bash` (e.g., macOS, BSD, NixOS).

**Alternative**: `#!/bin/bash` if you know the script only runs on Linux and prefer explicit paths.

### 2. Strict Mode
```bash
set -euo pipefail
```
**What each flag does:**
- `-e`: Exit immediately if any command fails (non-zero exit)
- `-u`: Treat unset variables as errors
- `-o pipefail`: Pipe fails if ANY command in pipeline fails (not just the last)

**When to add `-x`**: Only for debugging, not in production scripts (makes output noisy).

### 3. ShellCheck Compliance
Run ShellCheck on EVERY script before committing:
```bash
shellcheck script.sh
```

Fix all warnings. ShellCheck catches:
- Unquoted variables
- Deprecated syntax
- Common bugs and pitfalls
- Portability issues

### 4. Basic Script Structure
```bash
#!/usr/bin/env bash
set -euo pipefail

# Brief description of what this script does

# Simple error reporting
die() {
    echo "Error: ${1}" >&2
    exit 1
}

# Your code here
```

## Core Safety Patterns

### Always Quote Variables
**Why**: Prevents word splitting and globbing disasters.

```bash
# Wrong - dangerous
cp $source $destination
rm -rf $prefix/bin

# Correct - safe
cp "${source}" "${destination}"
rm -rf "${prefix}/bin"

# Special case: Always use braces with variables
echo "${var}"      # Good
echo "$var"        # Acceptable but less consistent
echo $var          # Bad - unquoted
```

### Check Required Variables
```bash
# Fail fast if required variables aren't set
: "${REQUIRED_VAR:?REQUIRED_VAR must be set}"

# Or with custom message
: "${DATABASE_URL:?DATABASE_URL is required. Set it in .env}"
```

### Validate Inputs
```bash
# Check file exists before operating on it
[[ -f "${config_file}" ]] || die "Config file not found: ${config_file}"

# Check command exists before using it
command -v jq >/dev/null 2>&1 || die "jq is required but not installed"

# Validate directory before cd
[[ -d "${target_dir}" ]] || die "Directory does not exist: ${target_dir}"
```

## Essential Patterns

### Pattern 1: Simple Script Template
Use this for straightforward scripts:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Description: Process log files and extract errors

die() {
    echo "Error: ${1}" >&2
    exit 1
}

# Check dependencies
command -v jq >/dev/null 2>&1 || die "jq required"

# Validate arguments
[[ $# -eq 1 ]] || die "Usage: ${0} <logfile>"
logfile="${1}"
[[ -f "${logfile}" ]] || die "File not found: ${logfile}"

# Main logic
grep ERROR "${logfile}" | jq -r '.message'
```

### Pattern 2: Cleanup on Exit
Use trap for guaranteed cleanup:

```bash
#!/usr/bin/env bash
set -euo pipefail

# Create temp directory and ensure cleanup
tmpdir=$(mktemp -d)
trap 'rm -rf "${tmpdir}"' EXIT

# Now use tmpdir safely - cleanup happens automatically
echo "Working in: ${tmpdir}"
```

### Pattern 3: Safe Function Definition
Functions should be simple and focused:

```bash
# Good: Simple, single-purpose function
check_dependency() {
    local cmd="${1}"
    command -v "${cmd}" >/dev/null 2>&1 || die "${cmd} not installed"
}

# Good: Local variables, clear purpose
process_file() {
    local file="${1}"
    local output="${2}"

    [[ -f "${file}" ]] || die "Input file missing: ${file}"

    # Do processing
    sed 's/foo/bar/g' "${file}" > "${output}"
}
```

**Important**: Declare and set variables from command substitution separately to catch errors:

```bash
# Wrong - hides errors
local result="$(failing_command)"

# Correct - catches errors
local result
result="$(failing_command)"  # Will fail properly with set -e
```

### Pattern 4: Safe Array Handling
Arrays are useful for handling lists with spaces:

```bash
# Create array
declare -a files=("file one.txt" "file two.txt" "file three.txt")

# Iterate safely - always quote with [@]
for file in "${files[@]}"; do
    echo "Processing: ${file}"
done

# Build command arguments safely
declare -a flags=(--verbose --output "${output_file}")
mycommand "${flags[@]}" "${input}"

# Read command output into array
mapfile -t lines < <(grep pattern "${file}")
```

### Pattern 5: Conditional Testing
Use `[[ ]]` for bash (safer and more features):

```bash
# File tests
[[ -f "${file}" ]]          # File exists
[[ -d "${dir}" ]]           # Directory exists
[[ -r "${file}" ]]          # File readable
[[ -w "${file}" ]]          # File writable
[[ -x "${binary}" ]]        # File executable

# String tests
[[ -z "${var}" ]]           # String is empty
[[ -n "${var}" ]]           # String is not empty
[[ "${a}" == "${b}" ]]      # String equality (use ==, not =)

# Numeric comparison (use (( )) for numbers)
(( count > 0 ))
(( total >= minimum ))

# Combined conditions
[[ -f "${file}" && -r "${file}" ]] || die "File not readable: ${file}"
```

### Pattern 6: Simple Argument Handling
For simple scripts, prefer positional arguments:

```bash
#!/usr/bin/env bash
set -euo pipefail

# For 1-3 arguments, just use positional parameters
[[ $# -eq 2 ]] || die "Usage: ${0} <source> <dest>"

source="${1}"
dest="${2}"

[[ -f "${source}" ]] || die "Source not found: ${source}"
```

For scripts needing flags, keep it simple:

```bash
# Use environment variables instead of complex flag parsing
VERBOSE="${VERBOSE:-false}"
DRY_RUN="${DRY_RUN:-false}"

# Run like: VERBOSE=true DRY_RUN=true ./script.sh input.txt
```

### Pattern 7: Process Substitution Over Temp Files
Avoid creating temporary files when possible:

```bash
# Instead of:
first_command > /tmp/output.txt
second_command < /tmp/output.txt
rm /tmp/output.txt

# Use process substitution:
second_command <(first_command)

# For multiple inputs:
diff <(sort file1.txt) <(sort file2.txt)
```

### Pattern 8: Prefer Builtins Over External Commands
Builtins are faster and more reliable:

```bash
# Use bash parameter expansion over sed/awk for simple cases
filename="${path##*/}"           # basename
dirname="${path%/*}"             # dirname
extension="${filename##*.}"      # get extension
name="${filename%.*}"            # remove extension

# Use (( )) for arithmetic over expr
count=$(( count + 1 ))           # Not: count=$(expr ${count} + 1)

# Use [[ ]] over [ ] or test
[[ -f "${file}" ]]               # Not: test -f "${file}"

# Use ${#var} for string length
length="${#string}"              # Not: length=$(echo "${string}" | wc -c)
```

## Intermediate Patterns

### Pattern 9: Structured Logging
Keep logging simple and consistent:

```bash
log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] ${1}" >&2
}

error() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: ${1}" >&2
}

# Usage
log "Starting process"
error "Failed to connect to database"
```

### Pattern 10: Main Function Pattern
For longer scripts (50+ lines), use a main function:

```bash
#!/usr/bin/env bash
set -euo pipefail

setup() {
    # Dependency checks, variable initialisation
    command -v jq >/dev/null 2>&1 || die "jq required"
}

process() {
    # Main logic here
    lo