Sysadmin

What this skill does

# System Administration Rules

## User Management

- Create service accounts with `--system` flag — no home directory, no login shell
- `sudo` with specific commands, not blanket ALL — principle of least privilege
- Lock accounts instead of deleting: `usermod -L` — preserves audit trail and file ownership
- SSH keys in `~/.ssh/authorized_keys` with restrictive permissions — 600 for file, 700 for directory
- `visudo` to edit sudoers — catches syntax errors before saving, prevents lockout

## Process Management

- `systemctl` for services, not `service` — systemd is standard on modern distros
- `journalctl -u service -f` for live logs — more powerful than tail on log files
- `nice` and `ionice` for background tasks — don't compete with production workloads
- Kill signals: SIGTERM (15) first, SIGKILL (9) last resort — SIGKILL doesn't allow cleanup
- `nohup` or `screen`/`tmux` for long-running commands — SSH disconnect kills regular processes

## File Systems and Storage

- `df -h` for disk usage, `du -sh *` to find culprits — check before disk fills completely
- `lsof +D /path` finds processes using a directory — needed before unmounting
- `ncdu` for interactive disk usage — faster than repeated du commands
- Mount options matter: `noexec`, `nosuid` for security on data partitions
- Resize filesystems with care: grow is safe, shrink risks data loss — always backup first

## Logs and Monitoring

- `logrotate` prevents disk fill — configure size limits and retention
- Centralize logs to external system — local logs lost if server dies
- `/var/log/auth.log` or `/var/log/secure` for login attempts — watch for brute force
- `dmesg` for kernel messages — hardware errors, OOM kills appear here
- Monitor inode usage, not just disk space — many small files exhaust inodes

## Permissions and Security

- `chmod 600` for secrets, `640` for configs, `644` for public — world-writable is almost never correct
- Sticky bit on shared directories (`chmod +t`) — users can only delete their own files
- `setfacl` for complex permissions — when traditional owner/group/other isn't enough
- `chattr +i` makes files immutable — even root can't modify without removing flag
- SELinux/AppArmor in enforcing mode — permissive logs but doesn't protect

## Package Management

- `apt update` before `apt upgrade` — upgrade without update uses stale package lists
- Unattended security updates: `unattended-upgrades` — critical patches shouldn't wait
- Pin package versions in production — unexpected upgrades cause unexpected outages
- Remove unused packages: `apt autoremove` — reduces attack surface and disk usage
- Know your package manager: apt/yum/dnf/pacman — commands differ, concepts similar

## Backups

- Test restores regularly — backups that can't restore are worthless
- Include package lists and configs, not just data — recreating environment is painful
- Offsite backups mandatory — local backups don't survive disk failure or ransomware
- Backup before any risky change — "I'll just quickly edit" famous last words
- Document restore procedure — 3am disaster is wrong time to figure it out

## Performance

- `top`/`htop` for live view, `vmstat` for trends — understand baseline before diagnosing
- `iotop` for disk I/O bottlenecks — slow disk often blamed on CPU
- Load average: 1.0 per core is healthy — consistently higher means queuing
- Swap usage isn't inherently bad — but consistent swapping indicates memory shortage
- `sar` for historical data — retroactively diagnose what happened during incident

## Networking Basics

- `ss -tulpn` shows listening ports — `netstat` is deprecated
- `ip addr` and `ip route` replace `ifconfig` and `route` — learn the new tools
- Check both host firewall and cloud security groups — traffic blocked at either level fails
- `/etc/hosts` for local overrides — quick testing without DNS changes
- `curl -v` shows full connection details — headers, timing, TLS handshake

## Common Mistakes

- Running services as root — one exploit owns the system
- No monitoring until something breaks — reactive is expensive
- Editing config without backup — `cp file file.bak` takes two seconds
- Rebooting to "fix" issues — masks the problem, it'll return
- Ignoring disk space warnings — 100% full causes cascading failures
- Forgetting timezone configuration — logs from different servers don't correlate