server-hardening

What this skill does

# Server Hardening — Production Linux Security

## Hardening Philosophy

Apply the **principle of least privilege** at every layer:
- Services run as dedicated non-root users
- Only required ports are open
- Only required packages are installed
- All privilege changes are audited
- System is kept patched automatically

---

## Automatic Security Updates

### unattended-upgrades (Ubuntu/Debian)

```bash
sudo apt install -y unattended-upgrades update-notifier-common apt-listchanges
sudo dpkg-reconfigure --priority=low unattended-upgrades
```

`/etc/apt/apt.conf.d/50unattended-upgrades`:
```
Unattended-Upgrade::Allowed-Origins {
    // Apply security updates from Ubuntu's security channel
    "${distro_id}:${distro_codename}-security";
    // Optional: ESM security updates (Ubuntu Pro)
    "UbuntuESMApps:${distro_codename}-apps-security";
    "UbuntuESM:${distro_codename}-infra-security";
};

// Automatically remove obsolete packages after upgrade
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";

// Reboot automatically if required (kernel/libc updates)
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "false";
Unattended-Upgrade::Automatic-Reboot-Time "03:00";

// Email notification on errors
Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::MailReport "on-change";

// Block packages that have failed for too long
Unattended-Upgrade::SyslogEnable "true";
Unattended-Upgrade::SyslogFacility "daemon";

// Split the upgrade into smaller batches to reduce memory pressure
Unattended-Upgrade::MinimalSteps "true";
```

`/etc/apt/apt.conf.d/20auto-upgrades`:
```
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
```

```bash
# Test dry run — verify which packages would be upgraded
sudo unattended-upgrades --dry-run --debug

# Check service status
sudo systemctl status unattended-upgrades

# View upgrade log
sudo cat /var/log/unattended-upgrades/unattended-upgrades.log
```

`needrestart` — prompts (or automatically restarts) services after library upgrades:
```bash
sudo apt install -y needrestart
# Configure automatic restart mode (no prompt in CI/automated contexts):
sudo sed -i "s/#\$nrconf{restart} = 'i';/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf
```

---

## SSH Hardening

`/etc/ssh/sshd_config.d/99-hardening.conf` (drop-in, overrides defaults):
```
# Disable root login entirely — use sudo from a named user account
PermitRootLogin no

# Disable password authentication — keys only
PasswordAuthentication no
ChallengeResponseAuthentication no

# Disable X11 forwarding unless required
X11Forwarding no

# Disable agent forwarding (prevents lateral movement via forwarded SSH agent)
AllowAgentForwarding no

# Limit login window (default 120s is excessive)
LoginGraceTime 30

# Maximum authentication attempts before disconnection
MaxAuthTries 3

# Restrict SSH to specific users or groups
AllowUsers deploy admin
# AllowGroups sshusers

# Use only strong algorithms
KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
Ciphers [email protected],[email protected],[email protected]
MACs [email protected],[email protected]

# Disable SSH protocol 1 (already default, but explicit is better)
Protocol 2

# Enable strict mode checking on key file permissions
StrictModes yes

# Log logins at verbose level (captures key fingerprints)
LogLevel VERBOSE

# Disable TCP port forwarding unless required
AllowTcpForwarding no
```

```bash
sudo sshd -t        # Test config before reloading
sudo systemctl reload sshd
```

---

## AppArmor (Ubuntu/Debian)

```bash
# Check status
sudo aa-status

# Put a profile in complain mode (log violations, do not enforce)
sudo aa-complain /usr/sbin/nginx

# Enforce a profile
sudo aa-enforce /usr/sbin/nginx

# Load all profiles in /etc/apparmor.d/
sudo apparmor_parser -r /etc/apparmor.d/

# View recent violations
sudo journalctl -k | grep apparmor | tail -30
# OR
sudo cat /var/log/kern.log | grep apparmor | tail -30
```

Custom AppArmor profile skeleton (`/etc/apparmor.d/usr.local.bin.myapp`):
```
#include <tunables/global>

/usr/local/bin/myapp {
  #include <abstractions/base>
  #include <abstractions/nameservice>

  # Read app files
  /opt/myapp/** r,

  # Write to log directory
  /var/log/myapp/ rw,
  /var/log/myapp/** rw,

  # Read config
  /etc/myapp/** r,

  # Network access (outbound only)
  network inet stream,

  # Deny everything else
  deny /** w,
}
```

---

## SELinux Basics (RHEL/CentOS/Fedora)

```bash
# Check enforcement status
getenforce         # Enforcing / Permissive / Disabled
sestatus           # Detailed status

# Temporarily set permissive (testing — does not persist reboot)
sudo setenforce 0

# Re-enable enforcement
sudo setenforce 1

# Persistent mode in /etc/selinux/config:
# SELINUX=enforcing

# Fix wrong file context (e.g., after moving files)
sudo restorecon -Rv /var/www/html/

# Generate allow rules from audit denials
sudo ausearch -m avc -ts recent | audit2allow -M mypolicy
sudo semodule -i mypolicy.pp

# View denials
sudo ausearch -m avc -ts recent
```

---

## auditd: System Call Auditing

```bash
sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd
```

`/etc/audit/rules.d/99-production.rules`:
```bash
# Delete all existing rules and start fresh
-D

# Set buffer size (increase if losing events during heavy load)
-b 8192

# Failure mode: 1=print to syslog, 2=panic
-f 1

# ── Authentication and Session Events ─────────────────────────────────────────
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

# Track sudo usage (execve of sudo binary)
-a always,exit -F arch=b64 -F path=/usr/bin/sudo -F perm=x -k sudo_usage
-a always,exit -F arch=b64 -F path=/usr/bin/su -F perm=x -k su_usage

# SSH login events
-w /var/log/auth.log -p wa -k auth_log
-w /etc/ssh/sshd_config -p wa -k sshd_config

# ── Privilege Escalation ───────────────────────────────────────────────────────
# Track setuid/setgid program execution
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k root_commands

# ── File System Changes ────────────────────────────────────────────────────────
# Monitor critical system files
-w /etc/cron.d/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
-w /etc/hosts -p wa -k hosts_file
-w /etc/hostname -p wa -k hostname_file

# Monitor module loading
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k kernel_modules

# Monitor mount operations
-a always,exit -F arch=b64 -S mount -k mounts

# ── Network Connections ────────────────────────────────────────────────────────
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_ipv4
-a always,exit -F arch=b64 -S socket -F a0=10 -k network_socket_ipv6

# ── Make Rules Immutable (requires reboot to change) ─────────────────────────
-e 2
```

```bash
sudo augenrules --load
# Verify rules loaded
sudo auditctl -l

# Search audit log
sudo ausearch -k sudo_usage -ts today
sudo ausearch -k identity -ts recent

# Generate summary report
sudo aureport --summary
sudo aureport --failed --summary
sudo aureport -au --summary   # Authentication failures
```

---

## PAM Login Security

### Account Lockout After Failed Attempts

```bash
sudo apt install -y libpam-faillock
```

`/etc/security/faillock.conf`:
```ini
# Lock account after 5 failed attempts
deny = 5

# Unlock after 15 minutes (900 seconds)
unlock_time = 900

# Count failures for this many seconds
fail_interval = 900

# Also lock root account
even_deny_root = true

# Ignore users with UID below this (system accounts)
admin_group = wheel
```

Add