systemd

What this skill does

# systemd — Unit Files and System Management

## Unit File Anatomy

systemd units live in:
- `/lib/systemd/system/` — package-installed units (do not edit directly)
- `/etc/systemd/system/` — administrator units; override package units with same name
- `/etc/systemd/system/<name>.service.d/` — drop-in overrides (preferred)

Unit files use INI-style sections. Key sections for a service:

```
[Unit]     — metadata, dependencies, ordering
[Service]  — what to run and how to manage it
[Install]  — how to enable/disable (which targets want this unit)
```

---

## Production Service Unit Template

Save as `/etc/systemd/system/myapp.service`:

```ini
[Unit]
Description=My Application Server
Documentation=https://docs.example.com/myapp
# Ordering: start after network is up and PostgreSQL is ready
After=network-online.target postgresql.service
Wants=network-online.target
# Hard dependency: if PostgreSQL stops, this unit also stops
Requires=postgresql.service

[Service]
# --- Process type ---
# simple: ExecStart is the main process; systemd tracks it directly
# forking: process daemonises (old-style); set PIDFile=
# notify: process signals systemd via sd_notify() when ready
# oneshot: for scripts that run and exit; combine with RemainAfterExit=yes
Type=notify

# --- Identity ---
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp

# --- Environment ---
# Load secrets from a file not checked into source control
EnvironmentFile=/etc/myapp/environment
# Inline env vars (for non-secret values)
Environment=NODE_ENV=production
Environment=PORT=3000

# --- Process lifecycle ---
ExecStartPre=/opt/myapp/scripts/pre-start.sh   # Validation / migration
ExecStart=/usr/bin/node /opt/myapp/dist/server.js
ExecStop=/bin/kill -SIGTERM $MAINPID           # Graceful shutdown signal
ExecStopPost=/opt/myapp/scripts/post-stop.sh  # Cleanup after stop
ExecReload=/bin/kill -SIGHUP $MAINPID          # Signal for config reload (if supported)

# --- Restart behaviour ---
Restart=on-failure         # Restart if process exits non-zero or is killed
RestartSec=5s              # Wait 5 seconds before restarting
StartLimitIntervalSec=60s  # Window for counting start attempts
StartLimitBurst=5          # Max 5 starts in 60s; unit enters failed state after

# --- Resource limits ---
LimitNOFILE=65535          # Open file descriptors (overrides /etc/security/limits.conf)
LimitNPROC=4096            # Max subprocesses
MemoryMax=2G               # OOM-kill process if it exceeds 2 GB (cgroup-based)
CPUQuota=200%              # Max 2 CPU cores (200% of one core = 2 cores)
TasksMax=512               # Max number of tasks (threads + processes)

# --- Output logging ---
StandardOutput=journal     # stdout → journal
StandardError=journal      # stderr → journal
SyslogIdentifier=myapp     # Tag in journal (default: unit name without .service)

# --- Security hardening ---
PrivateTmp=true            # Mount private /tmp and /var/tmp (other services can't see them)
NoNewPrivileges=true       # Prevent setuid/setgid; process can't gain more privileges
ProtectSystem=strict       # /usr, /boot, /etc are read-only
ProtectHome=true           # /home, /root, /run/user are inaccessible
ReadWritePaths=/var/lib/myapp /var/log/myapp  # Exceptions to ProtectSystem=strict
PrivateDevices=true        # No access to physical devices
ProtectKernelTunables=true # Block writes to /proc/sys
ProtectKernelModules=true  # Block kernel module loading
ProtectControlGroups=true  # Block cgroup manipulation
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX  # Restrict socket families
RestrictNamespaces=true    # Block namespace creation
SystemCallFilter=@system-service  # Whitelist common syscalls; block dangerous ones

# --- Timeout ---
TimeoutStartSec=30s        # Fail if not ready within 30s (for Type=notify)
TimeoutStopSec=30s         # Force-kill after 30s if graceful stop takes too long

[Install]
WantedBy=multi-user.target  # Enable via: systemctl enable myapp
```

---

## EnvironmentFile Pattern for Secrets

```bash
# /etc/myapp/environment (mode 600, owned by root or myapp user)
DATABASE_URL=postgresql://myapp:secret@localhost:5432/myapp_prod
SECRET_KEY=your-256-bit-random-key-here
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
REDIS_URL=redis://:password@localhost:6379/0
```

```bash
# Set correct permissions
sudo install -m 600 -o root -g root /dev/stdin /etc/myapp/environment <<'EOF'
DATABASE_URL=...
EOF

# Or for apps running as a specific user
sudo install -m 640 -o root -g myapp /dev/stdin /etc/myapp/environment <<'EOF'
DATABASE_URL=...
EOF
```

The `EnvironmentFile` path in the unit file reads each `KEY=VALUE` line. Lines starting with `#` are ignored. A leading `-` makes the file optional: `EnvironmentFile=-/etc/myapp/environment`.

---

## Timer Unit — Daily Backup Job

### /etc/systemd/system/backup-myapp.service

```ini
[Unit]
Description=Daily backup of myapp database
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=backup
Group=backup
EnvironmentFile=/etc/myapp/environment
ExecStart=/opt/myapp/scripts/backup.sh
StandardOutput=journal
StandardError=journal
SyslogIdentifier=backup-myapp
```

### /etc/systemd/system/backup-myapp.timer

```ini
[Unit]
Description=Run myapp backup daily at 02:30 UTC
Requires=backup-myapp.service

[Timer]
# OnCalendar format: DayOfWeek Year-Month-Day Hour:Minute:Second
# Shortcuts: minutely, hourly, daily, weekly, monthly, yearly, quarterly
OnCalendar=*-*-* 02:30:00    # Every day at 02:30 UTC
Persistent=true               # If timer was missed (machine off), run immediately on next boot
RandomizedDelaySec=10min      # Jitter to avoid thundering herd on multiple servers
AccuracySec=1min              # Allow 1 minute timing inaccuracy for better power management

[Install]
WantedBy=timers.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now backup-myapp.timer

# Verify timer status
sudo systemctl list-timers backup-myapp
sudo systemctl status backup-myapp.timer

# Trigger immediately (for testing)
sudo systemctl start backup-myapp.service
```

### OnCalendar Syntax Examples

```
OnCalendar=minutely              # Every minute
OnCalendar=hourly                # Every hour at :00
OnCalendar=daily                 # Every day at 00:00
OnCalendar=weekly                # Every Monday at 00:00
OnCalendar=monthly               # First day of month at 00:00
OnCalendar=*-*-* 02:30:00        # Every day at 02:30
OnCalendar=Mon *-*-* 03:00:00    # Every Monday at 03:00
OnCalendar=*-*-1 04:00:00        # First of every month at 04:00
OnCalendar=*-*-* *:0/15:00       # Every 15 minutes
OnCalendar=Mon..Fri *-*-* 09:00:00  # Weekdays at 09:00

# Verify calendar expression without executing
systemd-analyze calendar "Mon *-*-* 09:00:00"
```

---

## Drop-In Overrides

Override a package-provided unit without editing the original file:

```bash
# Opens editor; creates /etc/systemd/system/nginx.service.d/override.conf
sudo systemctl edit nginx

# Create override manually
sudo mkdir -p /etc/systemd/system/nginx.service.d/
sudo tee /etc/systemd/system/nginx.service.d/limits.conf > /dev/null <<'EOF'
[Service]
# Raise open file descriptor limit for Nginx
LimitNOFILE=65535
# Add environment variable
Environment=MALLOC_ARENA_MAX=2
EOF

sudo systemctl daemon-reload
sudo systemctl restart nginx

# Show effective merged unit (original + all drop-ins)
sudo systemctl cat nginx
```

Drop-in files in `<name>.service.d/` are applied alphabetically. Use numeric prefixes (e.g., `10-limits.conf`, `20-env.conf`) to control order.

---

## Socket Activation

systemd can hold a listening socket and pass it to the service on first connection. The service doesn't need to be running all the time.

### /etc/systemd/system/myapp.socket

```ini
[Unit]
Description=myapp socket activation

[Socket]
ListenStream=127.0.0.1:3000   # Or: ListenStream=/run/myapp/myapp.sock
Accept=false                   # false = pass socke