server-performance

What this skill does

# Server Performance Tuning — Production Linux

## Performance Tuning Checklist

Before tuning, always establish a baseline:
```bash
# Baseline snapshot
ss -s                              # Socket summary
cat /proc/sys/net/core/somaxconn   # Current connection backlog
ulimit -n                          # File descriptor limit (per-process)
cat /proc/sys/fs/file-max          # Global fd limit
free -h                            # Memory/swap
swapon --show                      # Active swap
cat /sys/block/sda/queue/scheduler # I/O scheduler
```

---

## Kernel / Network Tuning via sysctl

`/etc/sysctl.d/99-production.conf`:
```ini
# ── Connection Backlog ────────────────────────────────────────────────────────
# Maximum number of connections that can be queued for acceptance
# Default: 4096 (Ubuntu 22.04). Match this to Nginx worker_connections.
net.core.somaxconn = 65535

# Maximum number of packets in the kernel's receive queue per NIC
net.core.netdev_max_backlog = 65535

# Maximum number of SYN requests in the half-open connection queue
net.ipv4.tcp_max_syn_backlog = 65535

# ── TIME_WAIT Tuning ──────────────────────────────────────────────────────────
# Reduce TIME_WAIT timeout from 60s to 30s
# Note: Cannot go below ~15s safely (RFC recommends 2*MSL)
net.ipv4.tcp_fin_timeout = 30

# Allow reuse of TIME_WAIT sockets for new outgoing connections
# Safe for servers that are not load balancers
net.ipv4.tcp_tw_reuse = 1

# ── Ephemeral Ports ───────────────────────────────────────────────────────────
# Default range: 32768-60999. Expand for high-outbound-connection servers.
net.ipv4.ip_local_port_range = 1024 65535

# ── TCP Keepalive ─────────────────────────────────────────────────────────────
# Start keepalive probes after 60s idle (default: 7200s = 2 hours)
net.ipv4.tcp_keepalive_time = 60

# Interval between keepalive probes
net.ipv4.tcp_keepalive_intvl = 10

# Number of failed probes before declaring connection dead
net.ipv4.tcp_keepalive_probes = 6

# ── Memory ───────────────────────────────────────────────────────────────────
# Swappiness: 10 is appropriate for SSD-based servers with sufficient RAM.
# Set to 1 for Redis/memory-intensive services, NOT to 0 (may cause OOM issues).
vm.swappiness = 10

# dirty_ratio: % of RAM that can be dirty before processes are forced to write
# Reduce for databases to avoid sudden I/O stalls (default: 20)
vm.dirty_ratio = 10

# dirty_background_ratio: % at which background writeback starts (default: 5)
vm.dirty_background_ratio = 3

# ── File Descriptors ──────────────────────────────────────────────────────────
# Global kernel limit for open file handles
fs.file-max = 2097152

# ── Network Receive Buffers ───────────────────────────────────────────────────
# Increase receive/send buffer maximums for high-throughput servers
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

# ── Connection Tracking (if firewall/NAT in use) ──────────────────────────────
# Increase conntrack table size to avoid "nf_conntrack: table full" errors
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 300
```

Apply immediately:
```bash
sudo sysctl -p /etc/sysctl.d/99-production.conf
# Verify
sysctl net.core.somaxconn net.ipv4.tcp_fin_timeout vm.swappiness
```

---

## File Descriptor (ulimit) Configuration

### System-wide: `/etc/security/limits.conf`
```
# Syntax: <domain> <type> <item> <value>
# * applies to all users except root
*               soft    nofile          65535
*               hard    nofile          65535
root            soft    nofile          65535
root            hard    nofile          65535
www-data        soft    nofile          1048576
www-data        hard    nofile          1048576
postgres        soft    nofile          65535
postgres        hard    nofile          65535
```

Requires PAM `pam_limits.so` to be active (enabled by default on Ubuntu). Verify:
```bash
sudo -u www-data bash -c 'ulimit -n'
```

### Per-Service Systemd Override

`/etc/systemd/system/nginx.service.d/limits.conf`:
```ini
[Service]
LimitNOFILE=1048576
```

`/etc/systemd/system/postgresql.service.d/limits.conf`:
```ini
[Service]
LimitNOFILE=65535
LimitNPROC=65535
```

```bash
sudo systemctl daemon-reload
sudo systemctl restart nginx
# Verify nginx sees the new limit
cat /proc/$(pgrep -o nginx)/limits | grep 'open files'
```

---

## Nginx Performance Configuration

`/etc/nginx/nginx.conf` (worker and global section):
```nginx
# Match CPU core count; "auto" sets it automatically
worker_processes auto;

# Increase from default 1024. Match fs.file-max / worker_processes.
worker_rlimit_nofile 65535;

events {
    # Number of simultaneous connections per worker process
    # Total capacity: worker_processes * worker_connections
    worker_connections 16384;

    # Use epoll on Linux for efficient I/O multiplexing
    use epoll;

    # Accept multiple connections per epoll event (reduces syscall overhead)
    multi_accept on;
}

http {
    # Send file data directly from kernel buffer (zero-copy)
    sendfile on;

    # Bundle response headers with first data packet (requires sendfile)
    tcp_nopush on;

    # Reduce latency on small packets by disabling Nagle's algorithm
    tcp_nodelay on;

    # Keep connections open for 30 requests or 65 seconds, whichever comes first
    keepalive_timeout 65;
    keepalive_requests 1000;

    # Cache open file descriptors (avoids repeated open()/stat() calls)
    open_file_cache max=10000 inactive=30s;
    open_file_cache_valid 60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;

    # Compress text responses (substantial bandwidth saving)
    gzip on;
    gzip_comp_level 2;           # Level 2 is sweet spot for CPU vs ratio
    gzip_min_length 1024;
    gzip_types text/plain text/css text/javascript application/json
               application/javascript text/xml application/xml;

    # Upstream keepalive (for proxy_pass to app servers)
    upstream app_backend {
        server 127.0.0.1:3000;
        server 127.0.0.1:3001;
        keepalive 64;            # Persist 64 idle connections to backend
    }

    server {
        location / {
            proxy_pass http://app_backend;
            proxy_http_version 1.1;   # Required for keepalive to backend
            proxy_set_header Connection "";
        }
    }
}
```

Test and reload:
```bash
sudo nginx -t && sudo systemctl reload nginx
```

---

## PgBouncer: PostgreSQL Connection Pooling

Postgres creates a new OS process per connection (~5–10 MB RAM). At 500 connections that is 2.5–5 GB overhead before a single query runs. PgBouncer multiplexes client connections onto a small pool of real server connections.

### Pool Modes

| Mode | Description | Best For |
|------|-------------|----------|
| `session` | Server connection held for lifetime of client session | Legacy apps that use session-level state |
| `transaction` | Server connection returned to pool after each transaction | **Recommended** for most stateless web apps |
| `statement` | Server connection returned after each statement | Only when no multi-statement transactions used |

`/etc/pgbouncer/pgbouncer.ini`:
```ini
[databases]
# Format: <pgbouncer_db_name> = host=<pg_host> port=<pg_port> dbname=<real_db>
myapp = host=127.0.0.1 port=5432 dbname=myapp_production

[pgbouncer]
listen_addr = 127.0.0.1
listen_port = 6432

auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

pool_mode = transaction

# Maximum client connections PgBouncer will accept
max_client_conn = 1000

# Number of real Postgres connections per database/user pair
default_pool_size = 20

# Reserve pool for emergencies (used when default pool exhausted)
reserve_pool_size = 5

# After this many seconds, reserve pool kicks in
reserve_pool_timeout = 3

# Kill idle server connections after N seconds of inactivity
server_idle_timeout = 600

# Maximum age of server connection befo