debug-vps-deployment

What this skill does

# Debug VPS Deployment

Deploy to the production VPS ([email protected]) and iteratively debug failures until successful deployment.

## Objective

This skill connects to the specific VPS, triggers OtterStack deployments, monitors output for failures, diagnoses root causes, applies fixes, and redeploys until the application is successfully running and healthy.

## Orchestration Mode

This skill can be used in two modes:

1. **Standalone Mode** - Invoked directly by user for manual VPS debugging
2. **Orchestrated Mode** - Invoked automatically by `/deploy-otterstack` command during Phase 6: Debug Loop

### Orchestrated Mode Operation

When invoked from the `/deploy-otterstack` command, this skill operates in **iterative debug mode**:

**Workflow:**
1. Receives deployment error context from orchestration command
2. Diagnoses failure type using decision tree below
3. Applies appropriate fix (automatic or guided)
4. Returns fix status to orchestration command
5. Orchestration command retries deployment
6. Repeats until successful (max 6 attempts by default)

**Context Passed from Orchestration:**
- `ERROR_MESSAGE` - Full error output from failed deployment
- `DEPLOYMENT_STAGE` - Which stage failed (validation, startup, health_check, lock_conflict, traefik, unknown)
- `PROJECT_NAME` - Name of the project being deployed
- `ATTEMPT_NUMBER` - Current debug iteration (1-6)
- `DEPLOYMENT_TARGET` - Always "vps" when this skill is invoked

**Fix Determination:**
Based on `DEPLOYMENT_STAGE`, the skill uses the corresponding decision tree section below:
- **validation** → See "Failure Type 1: compose validation failed"
- **startup** → See "Failure Type 2: failed to start services"
- **health_check** → See "Failure Type 3: health check failed"
- **lock_conflict** → See "Failure Type 4: deployment lock already held"
- **traefik** → See "Failure Type 5: Traefik routing issues"
- **unknown** → Analyze full error output to identify type

**Fix Application:**

| Fix Type | Fixable Via | Orchestration Action |
|----------|-------------|----------------------|
| Missing env vars | SSH command | Auto-fix: `ssh ... otterstack env set PROJECT VAR value` |
| Stale lock file | SSH command | Auto-fix: `ssh ... rm ~/.otterstack/locks/PROJECT.lock` |
| Compose file issues | Local edit | Prompt user: "Edit compose file, then press Enter to retry" |
| Dockerfile issues | Local edit | Prompt user: "Edit Dockerfile, commit, push, then press Enter" |
| Code issues | Local edit | Prompt user: "Fix code, commit, push, then press Enter" |

**Return Values:**

After each debugging iteration, provide guidance for retry:

```
Fix applied: [yes/no/partial]
Fix type: [environment_variable/lock_file/compose_file/dockerfile/code/unknown]
Confidence: [high/medium/low]
Retry recommended: [yes/no]
Notes: [Brief description of what was fixed]
```

### Standalone vs Orchestrated Comparison

| Aspect | Standalone Mode | Orchestrated Mode |
|--------|-----------------|-------------------|
| Invocation | User runs skill directly | Called by /deploy-otterstack |
| Error Capture | Manual observation | Automatic from deployment output |
| Error Analysis | Manual diagnosis using decision tree | Automatic using DEPLOYMENT_STAGE |
| Fix Application | User decides when to fix | Semi-automated with prompts |
| Retry Logic | User manually redeploys | Automatic retry after fix |
| User Interaction | High - user drives process | Low - command drives, prompts for manual fixes |
| Iteration Limit | None - user decides | 6 attempts (configurable) |
| Context Awareness | User provides context | Full context from preparation phase |

### Integration Points

The orchestration command integrates this skill as follows:

**Before Invocation:**
```bash
# Parse deployment error
ERROR_MESSAGE=$(cat deployment_output.txt)
DEPLOYMENT_STAGE=$(parse_error_stage "$ERROR_MESSAGE")

# Invoke debug skill with context
# The skill analyzes the error and provides fix guidance
```

**During Debugging:**
```bash
# For auto-fixable issues (env vars, locks):
if [[ $FIX_TYPE == "environment_variable" ]]; then
  ssh [email protected] "${OTTERSTACK_PATH} env set ${PROJECT_NAME} ${VAR_NAME} '${VAR_VALUE}'"
fi

if [[ $FIX_TYPE == "lock_file" ]]; then
  ssh [email protected] "rm ~/.otterstack/locks/${PROJECT_NAME}.lock"
fi

# For manual fixes (compose, Dockerfile, code):
echo "Fix required: ${FIX_DESCRIPTION}"
read -p "Press Enter after fixing manually, or 'abort' to cancel: " response
[[ $response == "abort" ]] && exit 1
```

**After Fix:**
```bash
# Retry deployment
ssh [email protected] "~/OtterStack/otterstack deploy ${PROJECT_NAME} -v"

# If successful: proceed to Phase 7: Verification
# If failed: increment attempt counter, continue debug loop
```

## VPS Connection Details

```bash
# SSH Connection
SSH_HOST="[email protected]"
SSH_USER="archivist"
OTTERSTACK_PATH="~/OtterStack/otterstack"

# Verify connection
ssh ${SSH_HOST} "echo 'Connection successful'"
```

## Quick Start Deployment

```bash
# 1. Verify OtterStack is available
ssh ${SSH_HOST} "${OTTERSTACK_PATH} --help"

# 2. Check project exists
ssh ${SSH_HOST} "${OTTERSTACK_PATH} status <project-name>"

# 3. Deploy with verbose output
ssh ${SSH_HOST} "${OTTERSTACK_PATH} deploy <project-name> -v"

# 4. If it succeeds, verify endpoints. If it fails, proceed to debugging.
```

## Deployment Workflow

### Stage 1: Pre-Flight Checks

Before deploying, verify the environment is ready:

```bash
# Check SSH access
ssh ${SSH_HOST} "echo 'SSH OK'"

# Check OtterStack installation
ssh ${SSH_HOST} "${OTTERSTACK_PATH} --version" || \
  ssh ${SSH_HOST} "ls -l ~/OtterStack/otterstack"

# List existing projects
ssh ${SSH_HOST} "${OTTERSTACK_PATH} project list"

# Check current deployment status
ssh ${SSH_HOST} "${OTTERSTACK_PATH} status <project-name>"
```

### Stage 2: Trigger Deployment

Deploy with verbose output to see all stages:

```bash
ssh ${SSH_HOST} "${OTTERSTACK_PATH} deploy <project-name> -v"
```

### Stage 3: Monitor Deployment Stages

Watch the output for these sequential stages:

1. **"Fetching latest changes..."** → Git operations (only for remote repos)
2. **"Validating compose file..."** → Syntax and env var validation
3. **"Pulling images..."** → Docker image downloads
4. **"Starting services..."** → Container creation and startup
5. **"Waiting for containers to be healthy..."** → Health check polling
6. **"Applying Traefik priority labels..."** → Traffic routing setup (if Traefik enabled)
7. **"Deployment successful!"** → All done

If any stage fails, proceed to the corresponding debugging section below.

## Failure Diagnosis Decision Tree

### Failure Type 1: "compose validation failed"

**Symptoms:**
- Error during "Validating compose file..." stage
- Message like: `variable MYVAR is not set`
- Or: `services.web.image is undefined`

**Diagnosis Commands:**
```bash
# View full validation output
ssh ${SSH_HOST} "cd ~/.otterstack/repos/<project> && docker compose config"

# Check which env vars are set
ssh ${SSH_HOST} "${OTTERSTACK_PATH} env list <project-name>"

# View the env file being used
ssh ${SSH_HOST} "cat ~/.otterstack/envfiles/<project-name>.env"
```

**Common Causes:**
1. **Missing environment variables** → Variable used in compose file but not set
2. **Invalid YAML syntax** → Parse errors in compose file
3. **Undefined service references** → Service/network/volume doesn't exist

**Fix:**
```bash
# Add missing environment variables
ssh ${SSH_HOST} "${OTTERSTACK_PATH} env set <project> VAR value"

# For invalid syntax: fix compose file locally, commit, push, redeploy

# Verify fix
ssh ${SSH_HOST} "${OTTERSTACK_PATH} deploy <project-name> -v"
```

### Failure Type 2: "failed to start services"

**Symptoms:**
- Error during "Starting services..." stage
- Containers exit immediately
- Message like: `container exited with code 1`

**Diagnosis Commands:**
```bash
# OtterStack automatically shows last 50 lines on failure
# For more co