zero-downtime-deploy

What this skill does

# Zero-Downtime Deployment — Strategies and Automation

## Strategy Comparison

| Strategy | Traffic Shift | Rollback Speed | Cost | Risk |
|----------|--------------|----------------|------|------|
| Blue-Green | Instant (upstream or DNS swap) | Instant (swap back) | 2x infrastructure | Low — old version ready |
| Rolling | Gradual (instance by instance) | Medium (redeploy previous) | 1x infrastructure | Medium — mixed versions briefly |
| Canary | Percentage-based | Near-instant (drop to 0%) | 1x + small canary fleet | Low — small blast radius |

Choose based on: infrastructure cost tolerance, rollback requirements, and whether users can tolerate seeing mixed versions during deploy.

---

## Blue-Green Deployment

### Concept

Maintain two identical environments: **blue** (current live) and **green** (new version). Deploy to green, run smoke tests, then switch traffic from blue to green. Blue stays warm for instant rollback.

```
          ┌──────────┐
Users ──→ │  Nginx   │
          └─────┬────┘
                │ active_upstream.conf
         ┌──────┴──────┐
         ▼             ▼
  ┌─────────────┐  ┌─────────────┐
  │  Blue       │  │  Green      │
  │  :3000      │  │  :3001      │
  │  (old)      │  │  (new)      │
  └─────────────┘  └─────────────┘
```

### Nginx Configuration

`/etc/nginx/conf.d/upstreams.conf`:
```nginx
upstream blue {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream green {
    server 127.0.0.1:3001;
    keepalive 32;
}
```

`/etc/nginx/active_upstream.conf` (managed by deploy script):
```nginx
# This file is symlinked/replaced atomically by deploy scripts.
# Do not edit manually.
proxy_pass http://blue;
```

`/etc/nginx/sites-enabled/myapp.conf`:
```nginx
server {
    listen 80;
    server_name example.com;

    location / {
        include /etc/nginx/active_upstream.conf;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Health check configuration
        proxy_connect_timeout 5s;
        proxy_read_timeout 60s;
    }
}
```

### Blue-Green Upstream Swap Script

`/opt/scripts/blue-green-swap.sh`:
```bash
#!/bin/bash
# Blue-Green upstream swap for Nginx
# Usage: blue-green-swap.sh [blue|green]
set -euo pipefail

TARGET=${1:-}
ACTIVE_CONF="/etc/nginx/active_upstream.conf"
NGINX_CONF="/etc/nginx/nginx.conf"
LOG_FILE="/var/log/deploy/blue-green.log"
HEALTH_CHECK_TIMEOUT=30

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

# ── Determine current and target upstream ─────────────────────────────────────
current=$(grep -oP 'http://\K[a-z]+' "$ACTIVE_CONF" 2>/dev/null || echo "blue")
if [[ -z "$TARGET" ]]; then
    TARGET=$([ "$current" = "blue" ] && echo "green" || echo "blue")
fi

if [[ "$TARGET" != "blue" && "$TARGET" != "green" ]]; then
    echo "Usage: $0 [blue|green]" >&2
    exit 1
fi

log "Switching from $current to $TARGET"

# ── Determine target port ──────────────────────────────────────────────────────
TARGET_PORT=$([ "$TARGET" = "blue" ] && echo "3000" || echo "3001")

# ── Health check target before switching ──────────────────────────────────────
log "Health checking $TARGET on port $TARGET_PORT..."
deadline=$((SECONDS + HEALTH_CHECK_TIMEOUT))
while [[ $SECONDS -lt $deadline ]]; do
    if curl -fsS --max-time 3 "http://127.0.0.1:${TARGET_PORT}/health" > /dev/null 2>&1; then
        log "$TARGET is healthy"
        break
    fi
    log "Waiting for $TARGET health check..."
    sleep 2
done

if ! curl -fsS --max-time 3 "http://127.0.0.1:${TARGET_PORT}/health" > /dev/null 2>&1; then
    log "ERROR: $TARGET health check failed after ${HEALTH_CHECK_TIMEOUT}s — aborting swap"
    exit 1
fi

# ── Write new active_upstream.conf ────────────────────────────────────────────
# Write to temp file first, then atomically move
TMP_CONF=$(mktemp)
cat > "$TMP_CONF" <<EOF
# Active upstream: $TARGET (updated $(date '+%Y-%m-%d %H:%M:%S'))
proxy_pass http://${TARGET};
EOF
sudo mv "$TMP_CONF" "$ACTIVE_CONF"

# ── Test Nginx config and reload ──────────────────────────────────────────────
if ! sudo nginx -t 2>&1; then
    log "ERROR: Nginx config test failed — reverting"
    echo "proxy_pass http://${current};" | sudo tee "$ACTIVE_CONF" > /dev/null
    exit 1
fi

sudo nginx -s reload
log "Nginx reloaded — now serving from $TARGET"

# ── Final external health check ───────────────────────────────────────────────
sleep 2
if curl -fsS --max-time 5 "https://example.com/health" > /dev/null 2>&1; then
    log "External health check passed — deploy complete"
else
    log "WARNING: External health check failed — monitor closely"
fi

echo ""
echo "Deploy summary:"
echo "  Previous: $current"
echo "  Active:   $TARGET"
echo "  Port:     $TARGET_PORT"
echo ""
echo "To rollback: sudo /opt/scripts/blue-green-swap.sh $current"
```

```bash
sudo chmod +x /opt/scripts/blue-green-swap.sh
sudo mkdir -p /var/log/deploy

# Deploy to green and switch
./deploy-to-green.sh    # Deploy your app to port 3001
sudo /opt/scripts/blue-green-swap.sh green

# Instant rollback
sudo /opt/scripts/blue-green-swap.sh blue
```

---

## Rolling Deployment

Nginx upstream with `max_fails` for automatic routing around unhealthy instances:

```nginx
upstream app {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3002 max_fails=3 fail_timeout=30s;
    keepalive 32;
}
```

Rolling update loop:
```bash
#!/bin/bash
# Rolling update: restart each instance sequentially
APP_INSTANCES=(myapp@3000 myapp@3001 myapp@3002)
HEALTH_URL="http://127.0.0.1"

for INSTANCE in "${APP_INSTANCES[@]}"; do
    PORT=${INSTANCE##*@}
    echo "Restarting $INSTANCE..."
    sudo systemctl restart "$INSTANCE"

    echo "Waiting for $INSTANCE to become healthy..."
    for i in $(seq 1 30); do
        if curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
            echo "$INSTANCE is healthy after ${i}s"
            break
        fi
        sleep 1
    done

    if ! curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
        echo "ERROR: $INSTANCE failed health check — stopping rolling update"
        exit 1
    fi

    echo "$INSTANCE healthy — continuing to next instance..."
    sleep 2   # Brief pause between restarts
done

echo "Rolling update complete"
```

---

## Canary Deployment

Route a percentage of traffic to the new version using Nginx `split_clients`:

```nginx
# Hash on client IP for sticky routing (same user always sees same version)
split_clients "${remote_addr}" $upstream_pool {
    10%     canary;   # 10% to new version
    *       stable;   # 90% to current version
}

upstream stable {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream canary {
    server 127.0.0.1:3001;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://$upstream_pool;
    }
}
```

Adjust percentage by editing the `split_clients` block and reloading Nginx. Monitor error rate and latency in Grafana before increasing canary percentage. To promote canary to 100%: replace `stable` backend with canary code and set `*` to the stable upstream.

---

## Rsync + Symlink Atomic Deploy Script

### Directory Layout (Capistrano-style)

```
/var/www/myapp/
├── releases/
│   ├── 20241201_030000/    ← older release
│   ├── 20241202_030000/    ← previous release
│   └── 20241203_030000/    ← newest release
├── current -> releases/20241203_030000   ← symlink (atomic)
└── shared/
    ├── .env                ← never in repository
    ├── logs/ -> /var/log/myapp/
    └── uploads/
```

### Deploy Script

`/opt/scripts/deploy.sh`:
```bash
#!/bin/bash
# Rsync + atomic symlink deploy
# Usage: deploy.sh [--rollback]
set -euo pipefail

# ── Config ───────────────────────────────────────────────────────────────────